diff --git a/CHANGELOG.md b/CHANGELOG.md index 50af7422..815ea3ba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,17 @@ +### v0.4.2 +- update: allow duplicated name in device, check uniqueness of object_id ([#279](https://github.com/RWTH-EBC/FiLiP/pull/279)) +- add: validation for JEXL based expression ([#260](https://github.com/RWTH-EBC/FiLiP/pull/260)) +- add: tutorials for multi-entity ([#260](https://github.com/RWTH-EBC/FiLiP/pull/260)) +- add: add ``update_entity_relationships`` to allow relationship update ([#271](https://github.com/RWTH-EBC/FiLiP/pull/271)) +- add: timeseries query with all attrs and specific attr name ([#16](https://github.com/RWTH-EBC/FiLiP/pull/16)) +- add: flag to determine the deletion of registration when clearing the CB ([#267](https://github.com/RWTH-EBC/FiLiP/pull/267)) +- fix: rework tutorials for pydantic v2 ([#259](https://github.com/RWTH-EBC/FiLiP/pull/259)) +- fix: inconsistency of `entity_type` as required argument ([#188](https://github.com/RWTH-EBC/FiLiP/pull/188)) + ### v0.4.1 - fix: Session added as optional parameter to enable tls communication with clients ([#249](https://github.com/RWTH-EBC/FiLiP/pull/249)) +- fix: add missing package ``geojson_pydantic`` in setup.py ([#276](https://github.com/RWTH-EBC/FiLiP/pull/276)) +- add: support entity creation with keyvalues ([#264](https://github.com/RWTH-EBC/FiLiP/pull/264)) #### v0.4.0 - add tutorial for protected endpoint with bearer authentication ([#208](https://github.com/RWTH-EBC/FiLiP/issues/208)) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1d1e567e..bdc2aa04 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,44 +1,46 @@ # Contribute as a user The documentation, examples and tutorials should be understandable and the code bug-free. -As all user's have different backgrounds, you may not understand everything or encounter bugs. -In that case, PLEASE raise an issue [here](https://github. com/RWTH-EBC/filip/issues/new). +As all users have different backgrounds, you may not understand everything or encounter bugs. +In that case, PLEASE raise an issue [here](https://github.com/RWTH-EBC/filip/issues/new). -Consider labeling the issue using the flag `bug` or `documentation` / `question`. +Consider labeling the issue with an [appropriate label](https://github.com/RWTH-EBC/FiLiP/labels). # Contribute as a developer If you instead want to contribute new features or fix bugs yourself, we are more than happy. -Please also [raise an issue](https://github.com/RWTH-EBC/filip/issues/new) -create a new branch labeled `XY_some_name`. -Here, `XY` is the number of your issue and `some_name` is a meaningful -description. -Alternatively and preferred, issue branches are created automatically on issue -assignment with [robvanderleek/create-issue-branch](https://github.com/robvanderleek/create-issue-branch). +Please [raise an issue](https://github.com/RWTH-EBC/filip/issues/new). +Issue branches are created automatically on issue assignments. See [workflow definition](.github/workflows/issue-tracker.yml) and [configuration file](.github/issue-branch.yml) for customization. -Branch creation is skipped for issues with label "question". +Branch creation is skipped for issues with the label "question". -Once you're feature is ready, create a pull request and check if the pipeline succeeds. +Once your feature is ready, create a pull request and check if the pipeline succeeds. Assign a reviewer before merging. Once review is finished, you can merge. **Before** implementing or modifying modules, classes or functions, please read the following page. -## Styleguide +## Style guides -We use PEP8 as a styleguide. Some IDEs (like PyCharm) automatically show you code that is not in PEP8. If you don't have such an IDE, please read [this page](https://pep8.org/) to get a better understanding of it. +### Coding style guide + +We use PEP8 as a coding style guide. Some IDEs (like PyCharm) automatically show you code that is not in PEP8. If you don't have such an IDE, please read [this page](https://pep8.org/) to get a better understanding of it. + +### Committing style guide + +For committing style guide please use Conventional Commits 1.0.0. For more details how to structure your commits please visit this [page](https://www.conventionalcommits.org/en/v1.0.0/). ## Documentation -All created or modified function should be documented properly. +All created or modified functions should be documented properly. Try to follow the structure already present. -If possible, write a little doctest example into the docstring to make clear to user's what the desired output of your function is. +If possible, write a little doctest example into the docstring to make clear to the user what the desired output of your function is. All non-self-explanatory lines of code should include a comment. -Although you will notice that not all docstring are already in this style we use the google-style for docstrings, e.g. +Although you will notice that not all docstring are already in this style, we use the google-style for docstrings, e.g. ```python @@ -57,23 +59,25 @@ def foo(dummy: str , dummy2: Union[str, int]): Furthermore, we use type annotations as this helps users to automatically identify wrong usage of functions. -In a further step type annotations may also help to accelerate your code. -For further details please check the official [documentation on type hints](https://docs.python.org/3/library/typing.html). +In a further step, type annotations may also help to accelerate your code. +For more details please check the official [documentation on type hints](https://docs.python.org/3/library/typing.html). ## Unit-Tests -Espacially when creating new functions or classes, you have to add a unit-test function. -Open the `test_module.py` file in the `\tests`-directory and add a function to the class `TestModule`with a name like `test_my_new_function`. If you create a new module, you have to create a new `test_my_new_module.py` file and follow the existing structure of the -other test-files. +Especially when creating new functions or classes, you have to add a unit-test function. +Tests are located in the `\tests` directory. Every file that includes tests has a `test_` prefix. +Open the appropriate module where you want to write a test and add an appropriate function. +When you are adding tests to an existing test file, it is also recommended that you study the other tests in that file; it will teach you which precautions you have to take to make your tests robust and portable. +If the corresponding module does not exist, then you should create a new module with `test_` prefix and appropriate name. If you are not familiar with unit-tests, here is a quick summary: -- Test as many things as possible. Even seemingly silly tests like correct input-format help prevent future problems for new users -- use the `self.assertSOMETHING` functions provided by `unittest`. This way a test failure is presented correctly An error inside your test function will not be handeled as a failure but an error. -- If the success of your test depends on the used device, you can use decorators like `skip()`, `skipif(numpy.__version__<(1, 0), "not supported with your numpy version")`, etc. -- `setUp()` and `tearDown()` are called before and after each test. Use this functions to define parameters used in every test, or to close applications like Dymola once a test is completed. -- See the [unittest-documentation](https://docs.python.org/3/library/unittest.html#organizing-tests) for further information +- Test as many things as possible. Even seemingly silly tests like correct input-format help prevent future problems for new users. +- Use the `self.assertSOMETHING` functions provided by `unittest`. This way a test failure is presented correctly. An error inside your test function will not be handled as a failure but as an error. +- If the success of your test depends on the used development environment, you can use decorators like `skip()`, `skipif(numpy.__version__<(1, 0), "not supported with your numpy version")`, etc. +- `setUp()` and `tearDown()` are called before and after each test. Use these functions to define parameters used in every test, or to close applications like Dymola once a test is completed. +- See the [unittest-documentation](https://docs.python.org/3/library/unittest.html#organizing-tests) for further information. -You can check your work by running all tests before commiting to git. +You can check your work by running all tests before committing to git. ## Pylint With pylint we try to keep our code clean. -See the description in [this repo](https://git.rwth-aachen.de/EBC/EBC_all/gitlab_ci/templates/tree/master/pylint) on information on what pylint is and how to use it. +[Here](https://pypi.org/project/pylint/) you can read more about Pylint and how to use it. diff --git a/examples/README.md b/examples/README.md index f4b9ee30..bbc9bea5 100644 --- a/examples/README.md +++ b/examples/README.md @@ -36,5 +36,6 @@ The following topics are covered: #### How to use ontologies for semantic system modelling? -- [semantics](https://github.com/RWTH-EBC/FiLiP/tree/master/examples/ngsi_v2/e11_ngsi_v2_semantics) +- [Semantics](https://github.com/RWTH-EBC/FiLiP/tree/master/examples/ngsi_v2/e11_ngsi_v2_semantics) +- [Use-case specific data modeling](https://github.com/RWTH-EBC/FiLiP/tree/master/examples/ngsi_v2/e12_ngsi_v2_use_case_models.py) diff --git a/examples/ngsi_v2/e12_ngsi_v2_use_case_models.py b/examples/ngsi_v2/e12_ngsi_v2_use_case_models.py new file mode 100644 index 00000000..35e9ac81 --- /dev/null +++ b/examples/ngsi_v2/e12_ngsi_v2_use_case_models.py @@ -0,0 +1,156 @@ +""" +# This example shows a workflow, how you can define or reuse use case specific data +# models and ensure FIWARE compatibility by merging these models with existing data +# model in FiLiP. The merged models can be used for interaction with FIWARE platform +# and in other information processing systems to establish interoperability. + +# In short: this workflow shows you a way to keep use case model simple and +# reusable while ensuring the compatability with FIWARE NGSI-V2 standards +""" +from pydantic import ConfigDict, BaseModel +from pydantic.fields import Field, FieldInfo +from filip.models import FiwareHeader +from filip.models.ngsi_v2.context import ContextEntityKeyValues +from filip.clients.ngsi_v2.cb import ContextBrokerClient +from filip.utils.cleanup import clear_context_broker +from pprint import pprint + +# Host address of Context Broker +CB_URL = "http://localhost:1026" + +# You can here also change the used Fiware service +# FIWARE-Service +SERVICE = 'filip' +# FIWARE-Servicepath +SERVICE_PATH = '/' +fiware_header = FiwareHeader(service=SERVICE, + service_path=SERVICE_PATH) + + +# Reuse existing data model from the internet +class PostalAddress(BaseModel): + """ + https://schema.org/PostalAddress + """ + + model_config = ConfigDict(populate_by_name=True, coerce_numbers_to_str=True) + + address_country: str = Field( + alias="addressCountry", + description="County code according to ISO 3166-1-alpha-2", + ) + street_address: str = Field( + alias="streetAddress", + description="The street address. For example, 1600 Amphitheatre Pkwy.", + ) + address_region: str = Field( + alias="addressRegion", + default=None, + ) + address_locality: str = Field( + alias="addressLocality", + default=None, + description="The locality in which the street address is, and which is " + "in the region. For example, Mountain View.", + ) + postal_code: str = Field( + alias="postalCode", + default=None, + description="The postal code. For example, 94043.", + ) + + +# It is assumed that this kind of models exists in use case, which is simple and use case +# specific. It describes basically, how does the data look like in the specific use case. +class WeatherStation(BaseModel): + model_config = ConfigDict(coerce_numbers_to_str=True, extra="ignore") + temperature: float = Field(default=20.0) + humidity: float = Field(default=50.0) + pressure: float = Field(default=1.0) + address: PostalAddress + + +# Merge the use case model with the FIWARE simplified data model to ensure FIWARE +# compatibility. +class WeatherStationFIWARE(WeatherStation, ContextEntityKeyValues): + # add default for type if not explicitly set + type: str = FieldInfo.merge_field_infos( + # First position is the field info of the parent class + ContextEntityKeyValues.model_fields["type"], + # set the default value + default="CustomModels:WeatherStation", + # overwrite the title in the json-schema if necessary + title="Type of the Weather Station", + # overwrite the description + description="Type of the Weather Station", + # validate the default value if necessary + validate_default=True, + # freeze the field if necessary + frozen=True, + # for more options see the pydantic documentation + ) + + +if __name__ == "__main__": + # Now we can export both the use case model and the FIWARE specific + # models to json-schema files and share it with other stakeholders + # or applications/services that need to use the data. + use_case_model = WeatherStation.model_json_schema() + pprint(use_case_model) + + fiware_specific_model = WeatherStationFIWARE.model_json_schema() + pprint(fiware_specific_model) + + # Workflow to utilize these data models. + + # 0. Initial client + cb_client = ContextBrokerClient(url=CB_URL, + fiware_header=fiware_header) + # clear cb + clear_context_broker(cb_client=cb_client) + + # 1. Crate data + weather_station = WeatherStationFIWARE( + id="myWeatherStation", + type="WeatherStation", + temperature=20, + address={ + "address_country": "Germany", + "street_address": "Mathieustr. 10", + "postal_code": 52072, + }, + ) + cb_client.post_entity(entity=weather_station, key_values=True, + update=True) + + # 2. Update data + weather_station.temperature = 30 # represent use case algorithm + cb_client.update_entity_key_values(entity=weather_station) + + # 3. Query and validate data + # represent querying data by data users + weather_station_data = cb_client.get_entity(entity_id="myWeatherStation", + response_format="keyValues") + # validate with general model + weather_station_2_general = WeatherStation.model_validate( + weather_station_data.model_dump() + ) + # validate with fiware specific model + weather_station_2_fiware = WeatherStationFIWARE.model_validate( + weather_station_data.model_dump() + ) + + # 4. Use data for different purposes + # for use case specific usage + print("Data complied with general model can be forwarded to other platform/system:\n" + f"{weather_station_2_general.model_dump_json(indent=2)}") + print(f"For example, address still comply with existing model:\n" + f"{weather_station_2_general.address.model_dump_json(indent=2)}\n") + + # for fiware specific usage + print("For usage within FIWARE system, id and type is helpful, e.g. for creating" + "notification for entity:\n" + f"{weather_station_2_fiware.model_dump_json(indent=2, include={'id', 'type'})}\n") + + # clear cb + clear_context_broker(cb_client=cb_client) diff --git a/filip/__init__.py b/filip/__init__.py index 0916bc06..82b1b4d3 100644 --- a/filip/__init__.py +++ b/filip/__init__.py @@ -4,4 +4,4 @@ from filip.config import settings from filip.clients.ngsi_v2 import HttpClient -__version__ = '0.4.0' +__version__ = '0.4.1' diff --git a/filip/clients/ngsi_v2/cb.py b/filip/clients/ngsi_v2/cb.py index 1f640223..57b4998b 100644 --- a/filip/clients/ngsi_v2/cb.py +++ b/filip/clients/ngsi_v2/cb.py @@ -197,10 +197,11 @@ def get_statistics(self) -> Dict: # Entity Operations def post_entity( self, - entity: ContextEntity, + entity: Union[ContextEntity, ContextEntityKeyValues], update: bool = False, patch: bool = False, override_attr_metadata: bool = True, + key_values: bool = False, ): """ Function registers an Object with the NGSI Context Broker, @@ -215,7 +216,7 @@ def post_entity( patch argument. Args: - entity (ContextEntity): + entity (ContextEntity/ContextEntityKeyValues): Context Entity Object update (bool): If the response.status_code is 422, whether the override and @@ -227,13 +228,27 @@ def post_entity( Only applies for patch equal to `True`. Whether to override or append the attribute's metadata. `True` for overwrite or `False` for update/append - + key_values(bool): + By default False. If set to True, "options=keyValues" will + be included in params of post request. The payload uses + the keyValues simplified entity representation, i.e. + ContextEntityKeyValues. """ url = urljoin(self.base_url, "v2/entities") headers = self.headers.copy() + params = {} + options = [] + if key_values: + assert isinstance(entity, ContextEntityKeyValues) + options.append("keyValues") + else: + assert isinstance(entity, ContextEntity) + if options: + params.update({'options': ",".join(options)}) try: res = self.post( - url=url, headers=headers, json=entity.model_dump(exclude_none=True) + url=url, headers=headers, json=entity.model_dump(exclude_none=True), + params=params, ) if res.ok: self.logger.info("Entity successfully posted!") @@ -242,11 +257,14 @@ def post_entity( except requests.RequestException as err: if update and err.response.status_code == 422: return self.override_entity( - entity=entity) + entity=entity, key_values=key_values) if patch and err.response.status_code == 422: - return self.patch_entity( - entity=entity, override_attr_metadata=override_attr_metadata - ) + if not key_values: + return self.patch_entity( + entity=entity, override_attr_metadata=override_attr_metadata + ) + else: + return self.update_entity_key_values(entity=entity) msg = f"Could not post entity {entity.id}" self.log_error(err=err, msg=msg) raise @@ -528,6 +546,36 @@ def update_entity(self, entity: ContextEntity, append_strict: bool = False): The request payload is an object representing the attributes to append or update. + Note: + Update means overwriting the existing entity. If you want to + manipulate you should rather use patch_entity. + + Args: + entity (ContextEntity): + append_strict: If `False` the entity attributes are updated (if they + previously exist) or appended (if they don't previously exist) + with the ones in the payload. + If `True` all the attributes in the payload not + previously existing in the entity are appended. In addition + to that, in case some of the attributes in the payload + already exist in the entity, an error is returned. + More precisely this means a strict append procedure. + + Returns: + None + """ + self.update_or_append_entity_attributes( + entity_id=entity.id, + entity_type=entity.type, + attrs=entity.get_attributes(), + append_strict=append_strict, + ) + + def update_entity_properties(self, entity: ContextEntity, append_strict: bool = False): + """ + The request payload is an object representing the attributes, of any type + but Relationship, to append or update. + Note: Update means overwriting the existing entity. If you want to manipulate you should rather use patch_entity. @@ -553,10 +601,41 @@ def update_entity(self, entity: ContextEntity, append_strict: bool = False): append_strict=append_strict, ) + def update_entity_relationships(self, entity: ContextEntity, + append_strict: bool = False): + """ + The request payload is an object representing only the attributes, of type + Relationship, to append or update. + + Note: + Update means overwriting the existing entity. If you want to + manipulate you should rather use patch_entity. + + Args: + entity (ContextEntity): + append_strict: If `False` the entity attributes are updated (if they + previously exist) or appended (if they don't previously exist) + with the ones in the payload. + If `True` all the attributes in the payload not + previously existing in the entity are appended. In addition + to that, in case some of the attributes in the payload + already exist in the entity, an error is returned. + More precisely this means a strict append procedure. + + Returns: + None + """ + self.update_or_append_entity_attributes( + entity_id=entity.id, + entity_type=entity.type, + attrs=entity.get_relationships(), + append_strict=append_strict, + ) + def delete_entity( self, entity_id: str, - entity_type: str, + entity_type: str= None, delete_devices: bool = False, iota_client: IoTAClient = None, iota_url: AnyHttpUrl = settings.IOTA_URL, @@ -569,7 +648,8 @@ def delete_entity( entity_id: Id of the entity to be deleted entity_type: - several entities with the same entity id. + Entity type, to avoid ambiguity in case there are several + entities with the same entity id. delete_devices: If True, also delete all devices that reference this entity (entity_id as entity_name) @@ -585,8 +665,10 @@ def delete_entity( """ url = urljoin(self.base_url, f"v2/entities/{entity_id}") headers = self.headers.copy() - params = {"type": entity_type} - + if entity_type: + params = {'type': entity_type} + else: + params = None try: res = self.delete(url=url, params=params, headers=headers) if res.ok: @@ -616,10 +698,13 @@ def delete_entity( headers=self.headers, ) - for device in iota_client_local.get_device_list(entity_names=[entity_id]): - if device.entity_type == entity_type: + for device in iota_client_local.get_device_list( + entity_names=[entity_id]): + if entity_type: + if device.entity_type == entity_type: + iota_client_local.delete_device(device_id=device.device_id) + else: iota_client_local.delete_device(device_id=device.device_id) - iota_client_local.close() def delete_entities(self, entities: List[ContextEntity]) -> None: @@ -664,9 +749,9 @@ def delete_entities(self, entities: List[ContextEntity]) -> None: def update_or_append_entity_attributes( self, entity_id: str, - entity_type: str, attrs: List[Union[NamedContextAttribute, Dict[str, ContextAttribute]]], + entity_type: str = None, append_strict: bool = False, forcedUpdate: bool = False): """ @@ -705,6 +790,9 @@ def update_or_append_entity_attributes( params = {} if entity_type: params.update({'type': entity_type}) + else: + entity_type = "dummy" + options = [] if append_strict: options.append("append") @@ -740,12 +828,12 @@ def update_or_append_entity_attributes( self.log_error(err=err, msg=msg) raise - def update_entity_key_value(self, - entity: Union[ContextEntityKeyValues, dict],): + def update_entity_key_values(self, + entity: Union[ContextEntityKeyValues, dict],): """ The entity are updated with a ContextEntityKeyValues object or a dictionary contain the simplified entity data. This corresponds to a - 'PATcH' request. + 'PATCH' request. Only existing attribute can be updated! Args: @@ -777,11 +865,11 @@ def update_entity_key_value(self, self.log_error(err=err, msg=msg) raise - def update_entity_attributes_key_value(self, - entity_id: str, - attrs: dict, - entity_type: str = None, - ): + def update_entity_attributes_key_values(self, + entity_id: str, + attrs: dict, + entity_type: str = None, + ): """ Update entity with attributes in keyValues form. This corresponds to a 'PATcH' request. @@ -812,14 +900,14 @@ def update_entity_attributes_key_value(self, "type": entity_type }) entity = ContextEntityKeyValues(**entity_dict) - self.update_entity_key_value(entity=entity) + self.update_entity_key_values(entity=entity) def update_existing_entity_attributes( self, entity_id: str, - entity_type: str, attrs: List[Union[NamedContextAttribute, Dict[str, ContextAttribute]]], + entity_type: str = None, forcedUpdate: bool = False, override_metadata: bool = False ): @@ -847,7 +935,11 @@ def update_existing_entity_attributes( """ url = urljoin(self.base_url, f"v2/entities/{entity_id}/attrs") headers = self.headers.copy() - params = {"type": entity_type} + if entity_type: + params = {"type": entity_type} + else: + params = None + entity_type = "dummy" entity = ContextEntity(id=entity_id, type=entity_type) entity.add_attributes(attrs) @@ -879,7 +971,10 @@ def update_existing_entity_attributes( self.log_error(err=err, msg=msg) raise - def override_entity(self, entity: ContextEntity): + def override_entity(self, + entity: Union[ContextEntity, ContextEntityKeyValues], + **kwargs + ): """ The request payload is an object representing the attributes to override the existing entity. @@ -888,21 +983,25 @@ def override_entity(self, entity: ContextEntity): If you want to manipulate you should rather use patch_entity. Args: - entity (ContextEntity): + entity (ContextEntity or ContextEntityKeyValues): Returns: None """ - self.replace_entity_attributes(entity_id=entity.id, - entity_type=entity.type, - attrs=entity.get_properties()) + return self.replace_entity_attributes(entity_id=entity.id, + entity_type=entity.type, + attrs=entity.get_attributes(), + **kwargs + ) def replace_entity_attributes( self, entity_id: str, - entity_type: str, - attrs: List[Union[NamedContextAttribute, + attrs: Union[List[Union[NamedContextAttribute, Dict[str, ContextAttribute]]], - forcedUpdate: bool = False + Dict], + entity_type: str = None, + forcedUpdate: bool = False, + key_values: bool = False, ): """ The attributes previously existing in the entity are removed and @@ -913,11 +1012,17 @@ def replace_entity_attributes( entity_id: Entity id to be updated entity_type: Entity type, to avoid ambiguity in case there are several entities with the same entity id. - attrs: List of attributes to add to the entity + attrs: List of attributes to add to the entity or dict of + attributes in case of key_values=True. forcedUpdate: Update operation have to trigger any matching subscription, no matter if there is an actual attribute update or no instead of the default behavior, which is to updated only if attribute is effectively updated. + key_values(bool): + By default False. If set to True, "options=keyValues" will + be included in params of the request. The payload uses + the keyValues simplified entity representation, i.e. + ContextEntityKeyValues. Returns: None """ @@ -925,32 +1030,40 @@ def replace_entity_attributes( headers = self.headers.copy() params = {} options = [] + if entity_type: + params.update({"type": entity_type}) + else: + entity_type = "dummy" + if forcedUpdate: options.append("forcedUpdate") + + if key_values: + options.append("keyValues") + assert isinstance(attrs, dict) + else: + entity = ContextEntity(id=entity_id, type=entity_type) + entity.add_attributes(attrs) + attrs = entity.model_dump( + exclude={"id", "type"}, + exclude_none=True + ) if options: params.update({'options': ",".join(options)}) - if entity_type: - params.update({"type": entity_type}) - - entity = ContextEntity(id=entity_id, type=entity_type) - entity.add_attributes(attrs) try: res = self.put( url=url, headers=headers, - json=entity.model_dump( - exclude={"id", "type"}, - exclude_none=True - ), + json=attrs, params=params, ) if res.ok: - self.logger.info("Entity '%s' successfully " "updated!", entity.id) + self.logger.info("Entity '%s' successfully " "updated!", entity_id) else: res.raise_for_status() except requests.RequestException as err: - msg = f"Could not replace attribute of entity {entity.id} !" + msg = f"Could not replace attribute of entity {entity_id} !" self.log_error(err=err, msg=msg) raise @@ -1619,7 +1732,7 @@ def delete_registration(self, registration_id: str) -> None: # Batch operation API def update(self, *, - entities: List[ContextEntity], + entities: List[Union[ContextEntity, ContextEntityKeyValues]], action_type: Union[ActionType, str], update_format: str = None, forcedUpdate: bool = False, @@ -1676,13 +1789,13 @@ def update(self, options.append("overrideMetadata") if forcedUpdate: options.append("forcedUpdate") - if options: - params.update({'options': ",".join(options)}) if update_format: assert ( update_format == "keyValues" ), "Only 'keyValues' is allowed as update format" - params.update({"options": "keyValues"}) + options.append("keyValues") + if options: + params.update({'options': ",".join(options)}) update = Update(actionType=action_type, entities=entities) try: res = self.post( @@ -1794,8 +1907,8 @@ def post_command( self, *, entity_id: str, - entity_type: str, command: Union[Command, NamedCommand, Dict], + entity_type: str = None, command_name: str = None, ) -> None: """ diff --git a/filip/clients/ngsi_v2/quantumleap.py b/filip/clients/ngsi_v2/quantumleap.py index a2fa7114..f9471f83 100644 --- a/filip/clients/ngsi_v2/quantumleap.py +++ b/filip/clients/ngsi_v2/quantumleap.py @@ -5,7 +5,7 @@ import time from math import inf from collections import deque -from itertools import count +from itertools import count,chain from typing import Dict, List, Union, Deque, Optional from urllib.parse import urljoin import requests @@ -22,7 +22,7 @@ AttributeValues, \ TimeSeries, \ TimeSeriesHeader -from filip.utils.validators import validate_http_url + logger = logging.getLogger(__name__) @@ -1097,5 +1097,124 @@ def get_entity_attr_values_by_type(self, for new, old in zip(chunk, res): old.extend(new) - return res + + # v2/attrs + def get_entity_by_attrs(self, *, + entity_type: str = None, + from_date: str = None, + to_date: str = None, + limit: int = 10000, + offset: int = None + ) -> List[TimeSeries]: + """ + Get list of timeseries data grouped by each existing attribute name. + The timeseries data include all entities corresponding to each + attribute name as well as the index and values of this attribute in + this entity. + + Args: + entity_type (str): Comma-separated list of entity types whose data + are to be included in the response. Use only one (no comma) + when required. If used to resolve ambiguity for the given + entityId, make sure the given entityId exists for this + entityType. + from_date (str): The starting date and time (inclusive) from which + the context information is queried. Must be in ISO8601 format + (e.g., 2018-01-05T15:44:34) + to_date (str): The final date and time (inclusive) from which the + context information is queried. Must be in ISO8601 format + (e.g., 2018-01-05T15:44:34). + limit (int): Maximum number of results to be retrieved. + Default value : 10000 + offset (int): Offset for the results. + + Returns: + List of TimeSeriesEntities + """ + url = urljoin(self.base_url, 'v2/attrs') + res_q = self.__query_builder(url=url, + entity_type=entity_type, + from_date=from_date, + to_date=to_date, + limit=limit, + offset=offset) + first = res_q.popleft() + + res = chain.from_iterable(map(lambda x: self.transform_attr_response_model(x), + first.get("attrs"))) + for chunk in res_q: + chunk = chain.from_iterable(map(lambda x: self.transform_attr_response_model(x), + chunk.get("attrs"))) + + for new, old in zip(chunk, res): + old.extend(new) + + return list(res) + + # v2/attrs/{attr_name} + def get_entity_by_attr_name(self, *, + attr_name: str, + entity_type: str = None, + from_date: str = None, + to_date: str = None, + limit: int = 10000, + offset: int = None + ) -> List[TimeSeries]: + """ + Get list of all entities containing this attribute name, as well as + getting the index and values of this attribute in every corresponding + entity. + + Args: + attr_name (str): The attribute name in interest. + entity_type (str): Comma-separated list of entity types whose data + are to be included in the response. Use only one (no comma) + when required. If used to resolve ambiguity for the given + entityId, make sure the given entityId exists for this + entityType. + from_date (str): The starting date and time (inclusive) from which + the context information is queried. Must be in ISO8601 format + (e.g., 2018-01-05T15:44:34) + to_date (str): The final date and time (inclusive) from which the + context information is queried. Must be in ISO8601 format + (e.g., 2018-01-05T15:44:34). + limit (int): Maximum number of results to be retrieved. + Default value : 10000 + offset (int): Offset for the results. + + Returns: + List of TimeSeries + """ + url = urljoin(self.base_url, f'/v2/attrs/{attr_name}') + res_q = self.__query_builder(url=url, + entity_type=entity_type, + from_date=from_date, + to_date=to_date, + limit=limit, + offset=offset) + + first = res_q.popleft() + res = self.transform_attr_response_model(first) + + for chunk in res_q: + chunk = self.transform_attr_response_model(chunk) + for new, old in zip(chunk, res): + old.extend(new) + return list(res) + + def transform_attr_response_model(self, attr_response): + res = [] + attr_name = attr_response.get("attrName") + for entity_group in attr_response.get("types"): + timeseries = map(lambda entity: + TimeSeries(entityId=entity.get("entityId"), + entityType=entity_group.get("entityType"), + index=entity.get("index"), + attributes=[ + AttributeValues(attrName=attr_name, + values=entity.get("values"))] + ), + entity_group.get("entities")) + res.append(timeseries) + return chain.from_iterable(res) diff --git a/filip/models/ngsi_v2/context.py b/filip/models/ngsi_v2/context.py index fd0a3489..20d28145 100644 --- a/filip/models/ngsi_v2/context.py +++ b/filip/models/ngsi_v2/context.py @@ -63,6 +63,16 @@ class GetEntitiesOptions(str, Enum): ) +class PropertyFormat(str, Enum): + """ + Format to decide if properties of ContextEntity class are returned as + List of NamedContextAttributes or as Dict of ContextAttributes. + """ + + LIST = "list" + DICT = "dict" + + class ContextAttribute(BaseAttribute, BaseValueAttribute): """ Model for an attribute is represented by a JSON object with the following @@ -173,15 +183,15 @@ def __init__(self, id: str, type: Union[str, Enum] = None, **data): # This will result in usual behavior super().__init__(id=id, type=type, **data) + def get_attributes(self) -> dict: + """ + Get the attribute of the entity with the given name in + dict format -class PropertyFormat(str, Enum): - """ - Format to decide if properties of ContextEntity class are returned as - List of NamedContextAttributes or as Dict of ContextAttributes. - """ - - LIST = "list" - DICT = "dict" + Returns: + dict + """ + return self.model_dump(exclude={"id", "type"}) class ContextEntity(ContextEntityKeyValues): @@ -673,7 +683,7 @@ class Update(BaseModel): description="actionType, to specify the kind of update action to do: " "either append, appendStrict, update, delete, or replace. ", ) - entities: List[ContextEntity] = Field( + entities: List[Union[ContextEntity, ContextEntityKeyValues]] = Field( description="an array of entities, each entity specified using the " "JSON entity representation format " ) diff --git a/filip/models/ngsi_v2/iot.py b/filip/models/ngsi_v2/iot.py index 4f45c136..105b33b4 100644 --- a/filip/models/ngsi_v2/iot.py +++ b/filip/models/ngsi_v2/iot.py @@ -4,17 +4,19 @@ from __future__ import annotations import logging import itertools +import warnings from enum import Enum from typing import Any, Dict, Optional, List, Union import pytz -from pydantic import field_validator, ConfigDict, BaseModel, Field, AnyHttpUrl +from pydantic import field_validator, model_validator, ConfigDict, BaseModel, Field, AnyHttpUrl from filip.models.base import NgsiVersion, DataType from filip.models.ngsi_v2.base import \ BaseAttribute, \ BaseValueAttribute, \ BaseNameAttribute -from filip.utils.validators import validate_fiware_datatype_string_protect, \ - validate_fiware_datatype_standard +from filip.utils.validators import (validate_fiware_datatype_string_protect, validate_fiware_datatype_standard, + validate_jexl_expression, validate_device_expression_language, + validate_service_group_expression_language) logger = logging.getLogger() @@ -57,7 +59,7 @@ class IoTABaseAttribute(BaseAttribute, BaseNameAttribute): "expression based on a combination of the reported values. " "See the Expression Language definition for details " "(https://iotagent-node-lib.readthedocs.io/en/latest/" - "expressionLanguage/index.html)" + "api.html#expression-language-support)" ) entity_name: Optional[str] = Field( default=None, @@ -68,10 +70,10 @@ class IoTABaseAttribute(BaseAttribute, BaseNameAttribute): "configured with the entity_type attribute. If no type is " "configured, the device entity type is used instead. " "Entity names can be defined as expressions, using the " - "Expression Language definition. " + "Expression Language definition " "(https://iotagent-node-lib.readthedocs.io/en/latest/" - "expressionLanguage/index.html) Allowed characters " - "are the ones in the plain ASCII set, except the following " + "api.html#expression-language-support). Allowed characters are" + " the ones in the plain ASCII set, except the following " "ones: control characters, whitespace, &, ?, / and #.", max_length=256, min_length=1, @@ -219,7 +221,7 @@ def validate_cbHost(cls, value): Returns: timezone """ - return str(value) + return str(value) if value else value lazy: Optional[List[LazyDeviceAttribute]] = Field( default=[], desription="list of common lazy attributes of the device. For each " @@ -248,13 +250,13 @@ def validate_cbHost(cls, value): "IoT Agents to store information along with the devices " "in the Device Registry." ) - expressionLanguage: Optional[ExpressionLanguage] = Field( - default="legacy", + expressionLanguage: ExpressionLanguage = Field( + default=ExpressionLanguage.JEXL, description="optional boolean value, to set expression language used " "to compute expressions, possible values are: " - "legacy or jexl. When not set or wrongly set, legacy " - "is used as default value." + "legacy or jexl, but legacy is deprecated." ) + valid_expressionLanguage = field_validator("expressionLanguage")(validate_service_group_expression_language) explicitAttrs: Optional[bool] = Field( default=False, description="optional boolean value, to support selective ignore " @@ -319,13 +321,13 @@ class DeviceSettings(BaseModel): description="Name of the device transport protocol, for the IoT Agents " "with multiple transport protocols." ) - expressionLanguage: Optional[ExpressionLanguage] = Field( - default=None, + expressionLanguage: ExpressionLanguage = Field( + default=ExpressionLanguage.JEXL, description="optional boolean value, to set expression language used " "to compute expressions, possible values are: " - "legacy or jexl. When not set or wrongly set, legacy " - "is used as default value." + "legacy or jexl, but legacy is deprecated." ) + valid_expressionLanguage = field_validator("expressionLanguage")(validate_device_expression_language) explicitAttrs: Optional[bool] = Field( default=False, description="optional boolean value, to support selective ignore " @@ -415,6 +417,67 @@ def validate_timezone(cls, value): assert value in pytz.all_timezones return value + @model_validator(mode='after') + def validate_device_attributes_expression(self): + """ + Validates device attributes expressions based on the expression language (JEXL or Legacy, where Legacy is + deprecated). + + Args: + self: The Device instance. + + Returns: + The Device instance after validation. + """ + if self.expressionLanguage == ExpressionLanguage.JEXL: + for attribute in self.attributes: + if attribute.expression: + validate_jexl_expression(attribute.expression, attribute.name, self.device_id) + elif self.expressionLanguage == ExpressionLanguage.LEGACY: + warnings.warn(f"No validation for legacy expression language of Device {self.device_id}.") + + return self + + @model_validator(mode='after') + def validate_duplicated_device_attributes(self): + """ + Check whether device has identical attributes + Args: + self: dict of Device instance. + + Returns: + The dict of Device instance after validation. + """ + for i, attr in enumerate(self.attributes): + for other_attr in self.attributes[:i] + self.attributes[i + 1:]: + if attr.model_dump() == other_attr.model_dump(): + raise ValueError(f"Duplicated attributes found: {attr.name}") + return self + + @model_validator(mode='after') + def validate_device_attributes_name_object_id(self): + """ + Validate the device regarding the behavior with devices attributes. + According to https://iotagent-node-lib.readthedocs.io/en/latest/api.html and + based on our best practice, following rules are checked + - name is required, but not necessarily unique + - object_id is not required, if given must be unique, i.e. not equal to any + existing object_id and name + Args: + self: dict of Device instance. + + Returns: + The dict of Device instance after validation. + """ + for i, attr in enumerate(self.attributes): + for other_attr in self.attributes[:i] + self.attributes[i + 1:]: + if attr.object_id and other_attr.object_id and \ + attr.object_id == other_attr.object_id: + raise ValueError(f"object_id {attr.object_id} is not unique") + if attr.object_id and attr.object_id == other_attr.name: + raise ValueError(f"object_id {attr.object_id} is not unique") + return self + def get_attribute(self, attribute_name: str) -> Union[DeviceAttribute, LazyDeviceAttribute, StaticDeviceAttribute, @@ -456,7 +519,8 @@ def add_attribute(self, """ try: if type(attribute) == DeviceAttribute: - if attribute in self.attributes: + if attribute.model_dump(exclude_none=True) in \ + [attr.model_dump(exclude_none=True) for attr in self.attributes]: raise ValueError self.attributes.append(attribute) diff --git a/filip/utils/cleanup.py b/filip/utils/cleanup.py index bda995cc..1719f03d 100644 --- a/filip/utils/cleanup.py +++ b/filip/utils/cleanup.py @@ -1,6 +1,7 @@ """ Functions to clean up a tenant within a fiware based platform. """ +import warnings from functools import wraps from pydantic import AnyHttpUrl, AnyUrl @@ -13,9 +14,11 @@ QuantumLeapClient -def clear_context_broker(url: str = None, - fiware_header: FiwareHeader = None, - cb_client: ContextBrokerClient = None): +def clear_context_broker(url: str, + fiware_header: FiwareHeader, + clear_registrations: bool = False, + cb_client: ContextBrokerClient = None + ): """ Function deletes all entities, registrations and subscriptions for a given fiware header. To use TLS connection you need to provide the cb_client parameter @@ -28,8 +31,12 @@ def clear_context_broker(url: str = None, Args: url: Url of the context broker service fiware_header: header of the tenant - cb_client: enables TLS communication if created with Session object, only needed for self-signed certificates - + cb_client: enables TLS communication if created with Session object, only needed + for self-signed certificates + clear_registrations: Determines whether registrations should be deleted. + If registrations are deleted while devices with commands + still exist, these devices become unreachable. + Only set to true once such devices are cleared. Returns: None """ @@ -49,9 +56,10 @@ def clear_context_broker(url: str = None, assert len(client.get_subscription_list()) == 0 # clear registrations - for reg in client.get_registration_list(): - client.delete_registration(registration_id=reg.id) - assert len(client.get_registration_list()) == 0 + if clear_registrations: + for reg in client.get_registration_list(): + client.delete_registration(registration_id=reg.id) + assert len(client.get_registration_list()) == 0 def clear_iot_agent(url: Union[str, AnyHttpUrl] = None, @@ -63,7 +71,7 @@ def clear_iot_agent(url: Union[str, AnyHttpUrl] = None, as an argument with the Session object including the certificate and private key. Args: - url: Url of the context broker service + url: Url of the iot agent service fiware_header: header of the tenant iota_client: enables TLS communication if created with Session object, only needed for self-signed certificates diff --git a/filip/utils/validators.py b/filip/utils/validators.py index 1896272d..7f8ee218 100644 --- a/filip/utils/validators.py +++ b/filip/utils/validators.py @@ -3,12 +3,15 @@ """ import logging import re +import warnings from aenum import Enum from typing import Dict, Any, List from pydantic import AnyHttpUrl, validate_call from pydantic_core import PydanticCustomError from filip.custom_types import AnyMqttUrl - +from pyjexl.jexl import JEXL +from pyjexl.parser import Transform +from pyjexl.exceptions import ParseError logger = logging.getLogger(name=__name__) @@ -166,3 +169,67 @@ def validate_fiware_service_path(service_path): def validate_fiware_service(service): return match_regex(service, r"\w*$") + + +jexl_transformation_functions = { + "jsonparse": "(str) => JSON.parse(str)", + "jsonstringify": "(obj) => JSON.stringify(obj)", + "indexOf": "(val, char) => String(val).indexOf(char)", + "length": "(val) => String(val).length", + "trim": "(val) => String(val).trim()", + "substr": "(val, int1, int2) => String(val).substr(int1, int2)", + "addreduce": "(arr) => arr.reduce((i, v) => i + v)", + "lengtharray": "(arr) => len(arr)", + "typeof": "(val) => typeof val", + "isarray": "(arr) => Array.isArray(arr)", + "isnan": "(val) => isNaN(val)", + "parseint": "(val) => parseInt(val)", + "parsefloat": "(val) => parseFloat(val)", + "toisodate": "(val) => new Date(val).toISOString()", + "timeoffset": "(isostr) => new Date(isostr).getTimezoneOffset()", + "tostring": "(val) => str(val)", + "urlencode": "(val) => encodeURI(val)", + "urldecode": "(val) => decodeURI(val)", + "replacestr": "(str, from, to) => str.replace(from, to)", + "replaceregexp": "(str, reg, to) => str.replace(reg, to)", + "replaceallstr": "(str, from, to) => str.replace(from, to)", + "replaceallregexp": "(str, reg, to) => str.replace(reg, to)", + "split": "(str, ch) => str.split(ch)", + "mapper": "(val, values, choices) => choices[values.index(val)]", + "thmapper": "(val, values, choices) => choices[next((i for i, v in enumerate(values) if val <= v), None)]", + "bitwisemask": "(i, mask, op, shf) => ((int(i) & mask) if op == '&' else ((int(i) | mask) if op == '|' else ((int(i) ^ mask) if op == '^' else int(i))) >> shf)", + "slice": "(arr, init, end) => arr[init:end]", + "addset": "(arr, x) => list(set(arr).add(x))", + "removeset": "(arr, x) => list(set(arr).remove(x))", + "touppercase": "(val) => str(val).upper()", + "tolowercase": "(val) => str(val).lower()" +} + + +def validate_jexl_expression(expression, attribute_name, device_id): + try: + jexl_expression = JEXL().parse(expression) + if isinstance(jexl_expression, Transform): + if jexl_expression.name not in jexl_transformation_functions.keys(): + warnings.warn(f"{jexl_expression.name} might not supported") + except ParseError: + msg = f"Invalid JEXL expression '{expression}' inside the attribute '{attribute_name}' of Device '{device_id}'." + if '|' in expression: + msg += " If the expression contains the transform operator '|' you need to remove the spaces around it." + raise ParseError(msg) + return expression + + +def validate_device_expression_language(cls, expressionLanguage): + if expressionLanguage == "legacy": + warnings.warn(f"Using 'LEGACY' expression language inside {cls.__name__} is deprecated. Use 'JEXL' instead.") + + return expressionLanguage + + +def validate_service_group_expression_language(cls, expressionLanguage): + if expressionLanguage == "legacy": + warnings.warn(f"Using 'LEGACY' expression language inside {cls.__name__} is deprecated and does not work " + f"anymore, because each device uses 'JEXL' as default.") + + return expressionLanguage diff --git a/requirements.txt b/requirements.txt index b7c9d9f8..22d2e3eb 100644 --- a/requirements.txt +++ b/requirements.txt @@ -2,7 +2,7 @@ requests~=2.31.0 python-dotenv>=0.21.0 pydantic>=2.5.2,<2.7.0 pydantic-settings>=2.0.0,<2.3.0 -geojson_pydantic~=1.0.2 +geojson-pydantic~=1.0.2 aenum~=3.1.15 pathlib~=1.0.1 regex~=2023.10.3 @@ -18,6 +18,7 @@ wget~=3.2 stringcase~=1.2.0 paho-mqtt~=2.0.0 datamodel_code_generator[http]~=0.25.0 +pyjexl~=0.3.0 # tutorials matplotlib~=3.5.3; python_version < '3.9' matplotlib~=3.8.0; python_version >='3.9' diff --git a/setup.py b/setup.py index a87df433..892ed9ac 100644 --- a/setup.py +++ b/setup.py @@ -19,11 +19,13 @@ 'regex~=2023.10.3', 'requests~=2.31.0', 'rapidfuzz~=3.4.0', - 'wget~=3.2'] + 'geojson-pydantic~=1.0.2', + 'wget~=3.2', + 'pyjexl~=0.3.0'] SETUP_REQUIRES = INSTALL_REQUIRES.copy() -VERSION = '0.4.0' +VERSION = '0.4.1' setuptools.setup( name='filip', diff --git a/tests/clients/test_mqtt_client.py b/tests/clients/test_mqtt_client.py index 76a6f731..11166ffd 100644 --- a/tests/clients/test_mqtt_client.py +++ b/tests/clients/test_mqtt_client.py @@ -240,7 +240,7 @@ def on_command(client, obj, msg): entity_type=entity.type, command=context_command) - time.sleep(2) + time.sleep(5) # close the mqtt listening thread self.mqttc.loop_stop() # disconnect the mqtt device diff --git a/tests/clients/test_ngsi_v2_cb.py b/tests/clients/test_ngsi_v2_cb.py index 59b91318..a4b7fbf9 100644 --- a/tests/clients/test_ngsi_v2_cb.py +++ b/tests/clients/test_ngsi_v2_cb.py @@ -11,7 +11,7 @@ import paho.mqtt.client as mqtt from datetime import datetime, timedelta - +from urllib.parse import urlparse import requests from requests import RequestException from filip.models.base import FiwareHeader @@ -251,7 +251,7 @@ def test_entity_update(self): 1) append attribute 2) update existing attribute value 1) delete attribute - + Returns: """ @@ -261,10 +261,10 @@ def test_entity_update(self): entity_init = self.entity.model_copy(deep=True) attr_init = entity_init.get_attribute("temperature") attr_init.metadata = { - "metadata_init": { - "type": "Text", - "value": "something"} - } + "metadata_init": { + "type": "Text", + "value": "something"} + } attr_append = NamedContextAttribute(**{ "name": 'pressure', "type": 'Number', @@ -393,6 +393,62 @@ def test_entity_update(self): clear_all(fiware_header=self.fiware_header, cb_url=settings.CB_URL) + # 4) update only property or relationship + if "update_entity_properties" or "update_entity_relationship": + # post entity with a relationship attribute + entity_init = self.entity.model_copy(deep=True) + attrs = [ + NamedContextAttribute(name='in', type='Relationship', value='dummy1')] + entity_init.add_attributes(attrs=attrs) + client.post_entity(entity=entity_init, update=True) + + # create entity that differs in both attributes + entity_update = entity_init.model_copy(deep=True) + attrs = [NamedContextAttribute(name='temperature', + type='Number', + value=21), + NamedContextAttribute(name='in', type='Relationship', + value='dummy2')] + entity_update.update_attribute(attrs=attrs) + + # update only properties and compare + client.update_entity_properties(entity_update) + entity_db = client.get_entity(entity_update.id) + db_attrs = entity_db.get_attribute(attribute_name='temperature') + update_attrs = entity_update.get_attribute(attribute_name='temperature') + self.assertEqual(db_attrs, update_attrs) + db_attrs = entity_db.get_attribute(attribute_name='in') + update_attrs = entity_update.get_attribute(attribute_name='in') + self.assertNotEqual(db_attrs, update_attrs) + + # update only relationship and compare + attrs = [ + NamedContextAttribute(name='temperature', type='Number', value=22)] + entity_update.update_attribute(attrs=attrs) + client.update_entity_relationships(entity_update) + entity_db = client.get_entity(entity_update.id) + self.assertEqual(entity_db.get_attribute(attribute_name='in'), + entity_update.get_attribute(attribute_name='in')) + self.assertNotEqual(entity_db.get_attribute(attribute_name='temperature'), + entity_update.get_attribute( + attribute_name='temperature')) + + # change both, update both, compare + attrs = [NamedContextAttribute(name='temperature', + type='Number', + value=23), + NamedContextAttribute(name='in', type='Relationship', + value='dummy3')] + entity_update.update_attribute(attrs=attrs) + client.update_entity(entity_update) + entity_db = client.get_entity(entity_update.id) + db_attrs = entity_db.get_attribute(attribute_name='in') + update_attrs = entity_update.get_attribute(attribute_name='in') + self.assertEqual(db_attrs, update_attrs) + db_attrs = entity_db.get_attribute(attribute_name='temperature') + update_attrs = entity_update.get_attribute(attribute_name='temperature') + self.assertEqual(db_attrs, update_attrs) + @clean_test(fiware_service=settings.FIWARE_SERVICE, fiware_servicepath=settings.FIWARE_SERVICEPATH, cb_url=settings.CB_URL) @@ -680,11 +736,12 @@ def on_disconnect(client, userdata, flags, reasonCode, properties=None): mqtt_client.loop_start() new_value = 50 + time.sleep(1) self.client.update_attribute_value(entity_id=entity.id, attr_name='temperature', value=new_value, entity_type=entity.type) - time.sleep(5) + time.sleep(1) # test if the subscriptions arrives and the content aligns with updates self.assertIsNotNone(sub_message) @@ -694,6 +751,40 @@ def on_disconnect(client, userdata, flags, reasonCode, properties=None): mqtt_client.disconnect() time.sleep(1) + @clean_test(fiware_service=settings.FIWARE_SERVICE, + fiware_servicepath=settings.FIWARE_SERVICEPATH, + cb_url=settings.CB_URL) + def test_override_entity_keyvalues(self): + entity1 = self.entity.model_copy(deep=True) + # initial entity + self.client.post_entity(entity1) + + # entity with key value + entity1_key_value = self.client.get_entity( + entity_id=entity1.id, + response_format=AttrsFormat.KEY_VALUES) + + # override entity with ContextEntityKeyValues + entity1_key_value.temperature = 30 + self.client.override_entity(entity=entity1_key_value, key_values=True) + self.assertEqual(entity1_key_value, + self.client.get_entity( + entity_id=entity1.id, + response_format=AttrsFormat.KEY_VALUES) + ) + # test replace all attributes + entity1_key_value_dict = entity1_key_value.model_dump() + entity1_key_value_dict["temp"] = 40 + entity1_key_value_dict["humidity"] = 50 + self.client.override_entity( + entity=ContextEntityKeyValues(**entity1_key_value_dict), + key_values=True) + self.assertEqual(entity1_key_value_dict, + self.client.get_entity( + entity_id=entity1.id, + response_format=AttrsFormat.KEY_VALUES).model_dump() + ) + @clean_test(fiware_service=settings.FIWARE_SERVICE, fiware_servicepath=settings.FIWARE_SERVICEPATH, cb_url=settings.CB_URL) @@ -709,7 +800,7 @@ def test_update_entity_keyvalues(self): # update entity with ContextEntityKeyValues entity1_key_value.temperature = 30 - self.client.update_entity_key_value(entity=entity1_key_value) + self.client.update_entity_key_values(entity=entity1_key_value) self.assertEqual(entity1_key_value, self.client.get_entity( entity_id=entity1.id, @@ -722,10 +813,7 @@ def test_update_entity_keyvalues(self): # update entity with dictionary entity1_key_value_dict = entity1_key_value.model_dump() entity1_key_value_dict["temperature"] = 40 - self.client.update_entity_key_value(entity=entity1_key_value_dict) - self.client.get_entity( - entity_id=entity1.id, - response_format=AttrsFormat.KEY_VALUES).model_dump() + self.client.update_entity_key_values(entity=entity1_key_value_dict) self.assertEqual(entity1_key_value_dict, self.client.get_entity( entity_id=entity1.id, @@ -736,7 +824,7 @@ def test_update_entity_keyvalues(self): entity3.temperature.type) entity1_key_value_dict.update({"humidity": 50}) with self.assertRaises(RequestException): - self.client.update_entity_key_value(entity=entity1_key_value_dict) + self.client.update_entity_key_values(entity=entity1_key_value_dict) @clean_test(fiware_service=settings.FIWARE_SERVICE, fiware_servicepath=settings.FIWARE_SERVICEPATH, @@ -963,6 +1051,22 @@ def test_batch_operations(self): self.assertEqual(1000, len(client.query(query=query, response_format='keyValues'))) + # update with keyValues + entities_keyvalues = [ContextEntityKeyValues(id=str(i), + type=f'filip:object:TypeC', + attr1="text attribute", + attr2=1 + ) for i in range(0, 1000)] + client.update(entities=entities_keyvalues, + update_format="keyValues", + action_type=ActionType.APPEND) + entity_keyvalues = EntityPattern(idPattern=".*", typePattern=".*TypeC$") + query_keyvalues = Query.model_validate( + {"entities": [entity_keyvalues.model_dump(exclude_unset=True)]}) + entities_keyvalues_query = client.query(query=query_keyvalues, + response_format='keyValues') + self.assertEqual(1000, len(entities_keyvalues_query)) + self.assertEqual(1000, sum([e.attr2 for e in entities_keyvalues_query])) def test_force_update_option(self): """ @@ -1415,14 +1519,90 @@ def test_send_receive_string(self): self.client.post_entity(entity=entity) testData = "hello_test" - self.client.update_attribute_value(entity_id="string_test", attr_name="data", value=testData) + self.client.update_attribute_value(entity_id="string_test", attr_name="data", + value=testData) - readback = self.client.get_attribute_value(entity_id="string_test", attr_name="data") + readback = self.client.get_attribute_value(entity_id="string_test", + attr_name="data") self.assertEqual(testData, readback) self.client.delete_entity(entity_id="string_test", entity_type="test_type1") + def test_optional_entity_type(self): + """ + Test whether the entity type can be optional + """ + test_entity_id = "entity_type_test" + test_entity_type = "test1" + entity = ContextEntity(id=test_entity_id, type=test_entity_type) + entityAttr = NamedContextAttribute(name="data1", value="") + entity.add_attributes([entityAttr]) + self.client.post_entity(entity=entity) + + # test post_command + device_command = DeviceCommand(name='heater', type="Boolean") + device = Device(device_id='MyDevice', + entity_name='MyDevice', + entity_type='Thing', + protocol='IoTA-JSON', + transport='MQTT', + apikey=settings.FIWARE_SERVICEPATH.strip('/'), + commands=[device_command]) + self.iotac.post_device(device=device) + test_command = NamedCommand(name='heater', value=True) + self.client.post_command(entity_id="MyDevice", command=test_command) + + # update_or_append_entity_attributes + entityAttr.value = "value1" + attr_data2 = NamedContextAttribute(name="data2", value="value2") + self.client.update_or_append_entity_attributes(entity_id=test_entity_id, + attrs=[entityAttr, + attr_data2]) + + # update_existing_entity_attributes + self.client.update_existing_entity_attributes(entity_id=test_entity_id, + attrs=[entityAttr, + attr_data2]) + + # replace_entity_attributes + self.client.replace_entity_attributes(entity_id=test_entity_id, + attrs=[entityAttr, attr_data2]) + + # delete entity + self.client.delete_entity(entity_id=test_entity_id) + + # another entity with the same id but different type + test_entity_id_2 = "entity_type_test" + test_entity_type_2 = "test2" + entity_2 = ContextEntity(id=test_entity_id_2, type=test_entity_type_2) + self.client.post_entity(entity=entity_2) + self.client.post_entity(entity=entity) + + # update_or_append_entity_attributes + entityAttr.value = "value1" + attr_data2 = NamedContextAttribute(name="data2", value="value2") + with self.assertRaises(requests.HTTPError): + self.client.update_or_append_entity_attributes( + entity_id=test_entity_id, + attrs=[entityAttr, attr_data2]) + + # update_existing_entity_attributes + with self.assertRaises(requests.HTTPError): + self.client.update_existing_entity_attributes( + entity_id=test_entity_id, + attrs=[entityAttr, attr_data2]) + + # replace_entity_attributes + with self.assertRaises(requests.HTTPError): + self.client.replace_entity_attributes( + entity_id=test_entity_id, + attrs=[entityAttr, attr_data2]) + + # delete entity + with self.assertRaises(requests.HTTPError): + self.client.delete_entity(entity_id=test_entity_id) + def test_does_entity_exist(self): _id = uuid.uuid4() diff --git a/tests/clients/test_ngsi_v2_iota.py b/tests/clients/test_ngsi_v2_iota.py index 5300b46c..8518a285 100644 --- a/tests/clients/test_ngsi_v2_iota.py +++ b/tests/clients/test_ngsi_v2_iota.py @@ -18,8 +18,12 @@ DeviceAttribute, \ DeviceCommand, \ LazyDeviceAttribute, \ - StaticDeviceAttribute -from filip.utils.cleanup import clear_all, clean_test + StaticDeviceAttribute, ExpressionLanguage +from filip.utils.cleanup import \ + clear_all, \ + clean_test, \ + clear_context_broker, \ + clear_iot_agent from tests.config import settings logger = logging.getLogger(__name__) @@ -51,7 +55,7 @@ def setUp(self) -> None: "apikey": "1234", "endpoint": None, "transport": 'HTTP', - "expressionLanguage": None + "expressionLanguage": ExpressionLanguage.JEXL } self.client = IoTAClient( url=settings.IOTA_JSON_URL, @@ -430,11 +434,37 @@ def test_update_service_group(self): self.client.update_group(service_group=group_base) self.assertEqual(group_base, self.client.get_group(resource="/iot/json", apikey="base")) + @clean_test(fiware_service=settings.FIWARE_SERVICE, + fiware_servicepath=settings.FIWARE_SERVICEPATH, + iota_url=settings.IOTA_JSON_URL, + cb_url=settings.CB_URL) + def test_clear_iot_agent(self): + """ + Test for clearing iot agent AFTER clearing context broker + while having a device with a command + + Returns: + None + """ + cb_client = ContextBrokerClient(url=settings.CB_URL, + fiware_header=self.fiware_header) + device = Device(**self.device) + device.add_command(DeviceCommand(name="dummy_cmd")) + self.client.post_device(device=device) + clear_context_broker(settings.CB_URL, + self.fiware_header) + self.assertEqual(len(cb_client.get_registration_list()), 1) + + clear_iot_agent(settings.IOTA_JSON_URL, self.fiware_header) + self.assertCountEqual(cb_client.get_registration_list(), []) + def tearDown(self) -> None: """ Cleanup test server + """ self.client.close() clear_all(fiware_header=self.fiware_header, cb_url=settings.CB_URL, iota_url=settings.IOTA_JSON_URL) + diff --git a/tests/clients/test_ngsi_v2_timeseries.py b/tests/clients/test_ngsi_v2_timeseries.py index 8b2c206f..d5243431 100644 --- a/tests/clients/test_ngsi_v2_timeseries.py +++ b/tests/clients/test_ngsi_v2_timeseries.py @@ -387,6 +387,36 @@ def test_test_query_endpoints_with_args(self) -> None: self.assertGreater(old_records.index[0], records.index[0]) old_records = records + + def test_attr_endpoints(self) -> None: + """ + Test get entity by attr/attr name endpoints + Returns: + None + """ + with QuantumLeapClient( + url=settings.QL_URL, + fiware_header=FiwareHeader(service='filip', + service_path="/static")) \ + as client: + attr_names = ['temperature', 'humidity', 'co2'] + for attr_name in attr_names: + entities_by_attr_name = client.get_entity_by_attr_name( + attr_name=attr_name) + # we expect as many timeseries as there are unique ids + self.assertEqual(len(entities_by_attr_name), 2) + + # we expect the sizes of the index and attribute values to be the same + for timeseries in entities_by_attr_name: + for attribute in timeseries.attributes: + self.assertEqual(len(attribute.values), len(timeseries.index)) + + entities_by_attr = client.get_entity_by_attrs() + # we expect as many timeseries as : n of unique ids x n of different attrs + self.assertEqual(len(entities_by_attr), 2*3) + for timeseries in entities_by_attr: + for attribute in timeseries.attributes: + self.assertEqual(len(attribute.values), len(timeseries.index)) def tearDown(self) -> None: """ diff --git a/tests/models/test_ngsi_v2_iot.py b/tests/models/test_ngsi_v2_iot.py index b78e28c8..5bb11544 100644 --- a/tests/models/test_ngsi_v2_iot.py +++ b/tests/models/test_ngsi_v2_iot.py @@ -1,12 +1,20 @@ """ Test module for context broker models """ - +import time import unittest from typing import List +import warnings +from paho.mqtt import client as mqtt_client +import pyjexl +from filip.models.base import FiwareHeader from filip.models.ngsi_v2.iot import DeviceCommand, ServiceGroup, \ - Device, TransportProtocol, IoTABaseAttribute + Device, TransportProtocol, IoTABaseAttribute, ExpressionLanguage, PayloadProtocol, DeviceAttribute +from filip.clients.ngsi_v2 import ContextBrokerClient, IoTAClient + +from filip.utils.cleanup import clear_all, clean_test +from tests.config import settings class TestContextv2IoTModels(unittest.TestCase): @@ -21,6 +29,19 @@ def setUp(self) -> None: None """ + self.fiware_header = FiwareHeader( + service=settings.FIWARE_SERVICE, + service_path=settings.FIWARE_SERVICEPATH) + clear_all(fiware_header=self.fiware_header, + cb_url=settings.CB_URL, + iota_url=settings.IOTA_JSON_URL) + self.iota_client = IoTAClient( + url=settings.IOTA_JSON_URL, + fiware_header=self.fiware_header) + self.cb_client = ContextBrokerClient( + url=settings.CB_URL, + fiware_header=self.fiware_header) + def test_fiware_safe_fields(self): """ Tests all fields of models/ngsi_v2/iot.py that have a regex to @@ -29,7 +50,7 @@ def test_fiware_safe_fields(self): None """ - from pydantic.error_wrappers import ValidationError + from pydantic import ValidationError valid_strings: List[str] = ["name", "test123", "3_:strange-Name!"] invalid_strings: List[str] = ["my name", "Test?", "#False", "/notvalid"] @@ -94,4 +115,247 @@ def test_fiware_safe_fields(self): entity_name=string, entity_type=string) ServiceGroup(entity_type=string, resource="", apikey="") Device(device_id="", entity_name=string, entity_type=string, - transport=TransportProtocol.HTTP) \ No newline at end of file + transport=TransportProtocol.HTTP) + + @clean_test(fiware_service=settings.FIWARE_SERVICE, + fiware_servicepath=settings.FIWARE_SERVICEPATH, + cb_url=settings.CB_URL, + iota_url=settings.IOTA_JSON_URL) + def test_expression_language(self): + api_key = settings.FIWARE_SERVICEPATH.strip('/') + + # Test expression language on service group level + service_group_jexl = ServiceGroup( + entity_type='Thing', + apikey=api_key, + resource='/iot/json', + expressionLanguage=ExpressionLanguage.JEXL) + + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter("always") + + service_group_legacy = ServiceGroup( + entity_type='Thing', + apikey=api_key, + resource='/iot/json', + expressionLanguage=ExpressionLanguage.LEGACY) + + assert len(w) == 1 + assert issubclass(w[0].category, UserWarning) + assert "deprecated" in str(w[0].message) + + self.iota_client.post_group(service_group=service_group_jexl) + + # Test jexl expression language on device level + device1 = Device(device_id="test_device", + entity_name="test_entity", + entity_type="test_entity_type", + transport=TransportProtocol.MQTT, + protocol=PayloadProtocol.IOTA_JSON, + expressionLanguage=ExpressionLanguage.JEXL, + attributes=[DeviceAttribute(name="value", type="Number"), + DeviceAttribute(name="fraction", type="Number", + expression="(value + 3) / 10"), + DeviceAttribute(name="spaces", type="Text"), + DeviceAttribute(name="trimmed", type="Text", + expression="spaces|trim"), + ] + ) + self.iota_client.post_device(device=device1) + + mqtt_cl = mqtt_client.Client() + mqtt_cl.connect(settings.MQTT_BROKER_URL.host, settings.MQTT_BROKER_URL.port) + mqtt_cl.loop_start() + + mqtt_cl.publish(topic=f'/json/{api_key}/{device1.device_id}/attrs', + payload='{"value": 12, "spaces": " foobar "}') + + time.sleep(2) + + entity1 = self.cb_client.get_entity(entity_id=device1.entity_name) + self.assertEqual(entity1.get_attribute('fraction').value, 1.5) + self.assertEqual(entity1.get_attribute('trimmed').value, 'foobar') + + mqtt_cl.loop_stop() + mqtt_cl.disconnect() + + # Test for wrong jexl expressions + device2 = Device(device_id="wrong_device", + entity_name="test_entity", + entity_type="test_entity_type", + transport=TransportProtocol.MQTT, + protocol=PayloadProtocol.IOTA_JSON) + + attr1 = DeviceAttribute(name="value", type="Number", expression="value ++ 3") + with self.assertRaises(pyjexl.jexl.ParseError): + device2.add_attribute(attr1) + device2.delete_attribute(attr1) + + attr2 = DeviceAttribute(name="spaces", type="Text", expression="spaces | trim") + with self.assertRaises(pyjexl.jexl.ParseError): + device2.add_attribute(attr2) + device2.delete_attribute(attr2) + + attr3 = DeviceAttribute(name="brackets", type="Number", expression="((2 + 3) / 10") + with self.assertRaises(pyjexl.jexl.ParseError): + device2.add_attribute(attr3) + device2.delete_attribute(attr3) + + # Test for legacy expression warning + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter("always") + + device3 = Device(device_id="legacy_device", + entity_name="test_entity", + entity_type="test_entity_type", + transport=TransportProtocol.MQTT, + protocol=PayloadProtocol.IOTA_JSON, + expressionLanguage=ExpressionLanguage.LEGACY) + + assert len(w) == 2 + + def test_add_device_attributes(self): + """ + Test the device model regarding the behavior with devices attributes. + According to https://iotagent-node-lib.readthedocs.io/en/latest/api.html and + based on our best practice, following rules are checked + - name is required, but not necessarily unique + - object_id is not required, if given must be unique, i.e. not equal to any + existing object_id and name + """ + def initial_device(): + attr = DeviceAttribute( + name="temperature", + type="Number" + ) + return Device( + device_id="dummy:01", + entity_name="entity:01", + entity_type="MyEntity", + attributes=[attr] + ) + + # fail, because attr1 and attr ara identical + device_a = initial_device() + attr1 = DeviceAttribute( + name="temperature", + type="Number" + ) + with self.assertRaises(ValueError): + device_a.add_attribute(attribute=attr1) + + # fail, because the object_id is duplicated with the name of attr1 + device_b = initial_device() + attr2 = DeviceAttribute( + name="temperature", + type="Number", + object_id="temperature" + ) + with self.assertRaises(ValueError): + device_b.add_attribute(attribute=attr2) + + # success + device_c = initial_device() + attr3 = DeviceAttribute( + name="temperature", + type="Number", + object_id="t1" + ) + device_c.add_attribute(attribute=attr3) + # success + attr4 = DeviceAttribute( + name="temperature", + type="Number", + object_id="t2" + ) + device_c.add_attribute(attribute=attr4) + # fail, because object id is duplicated + attr5 = DeviceAttribute( + name="temperature2", + type="Number", + object_id="t2" + ) + with self.assertRaises(ValueError): + device_c.add_attribute(attribute=attr5) + + def test_device_creation(self): + """ + Test the device model regarding the behavior with devices attributes while + creating the devices. + According to https://iotagent-node-lib.readthedocs.io/en/latest/api.html and + based on our best practice, following rules are checked + - name is required, but not necessarily unique + - object_id is not required, if given must be unique, i.e. not equal to any + existing object_id and name + """ + + def create_device(attr1_name, attr2_name, + attr1_object_id=None, attr2_object_id=None): + _attr1 = DeviceAttribute( + name=attr1_name, + object_id=attr1_object_id, + type="Number" + ) + _attr2 = DeviceAttribute( + name=attr2_name, + object_id=attr2_object_id, + type="Number" + ) + return Device( + device_id="dummy:01", + entity_name="entity:01", + entity_type="MyEntity", + attributes=[_attr1, _attr2] + ) + + # fail, because attr1 and attr ara identical + with self.assertRaises(ValueError): + create_device( + attr1_name="temperature", + attr2_name="temperature", + attr1_object_id=None, + attr2_object_id=None + ) + + # fail, because the object_id is duplicated with the name of attr1 + with self.assertRaises(ValueError): + create_device( + attr1_name="temperature", + attr2_name="temperature", + attr1_object_id=None, + attr2_object_id="temperature" + ) + + # success + device = create_device( + attr1_name="temperature", + attr2_name="temperature", + attr1_object_id=None, + attr2_object_id="t1" + ) + # success + attr4 = DeviceAttribute( + name="temperature", + type="Number", + object_id="t2" + ) + device.add_attribute(attribute=attr4) + + # fail, because object id is duplicated + with self.assertRaises(ValueError): + create_device( + attr1_name="temperature2", + attr2_name="temperature", + attr1_object_id="t", + attr2_object_id="t" + ) + + def tearDown(self) -> None: + """ + Cleanup test server + """ + clear_all(fiware_header=self.fiware_header, + cb_url=settings.CB_URL, + iota_url=settings.IOTA_JSON_URL) + self.iota_client.close() + self.cb_client.close() diff --git a/tutorials/ngsi_v2/e1_virtual_weatherstation/e1_virtual_weatherstation.py b/tutorials/ngsi_v2/e1_virtual_weatherstation/e1_virtual_weatherstation.py index 33d2daab..3d4afef1 100644 --- a/tutorials/ngsi_v2/e1_virtual_weatherstation/e1_virtual_weatherstation.py +++ b/tutorials/ngsi_v2/e1_virtual_weatherstation/e1_virtual_weatherstation.py @@ -3,7 +3,7 @@ # Create a virtual IoT device that simulates the ambient temperature and # publishes it via MQTT. The simulation function is already predefined. -# This exercise to give a simple introduction to the communication via MQTT. +# This exercise gives a simple introduction to the communication via MQTT. # The input sections are marked with 'ToDo' @@ -31,16 +31,15 @@ # import simulation model from tutorials.ngsi_v2.simulation_model import SimulationModel - # ## Parameters -# ToDo: Enter your mqtt broker url and port, e.g mqtt://test.mosquitto.org:1883 +# ToDo: Enter your mqtt broker url and port, e.g. mqtt://test.mosquitto.org:1883. MQTT_BROKER_URL = "mqtt://test.mosquitto.org:1883" -# ToDo: If required enter your username and password +# ToDo: If required, enter your username and password. MQTT_USER = "" -MQTT_PW = "" +MQTT_PW = "" # ToDo: Create a unique topic that your weather station will publish on, -# e.g. by using a uuid +# e.g. by using a uuid. UNIQUE_ID = str(uuid4()) TOPIC_WEATHER_STATION = f"fiware_workshop/{UNIQUE_ID}/weather_station" @@ -50,8 +49,7 @@ T_SIM_START = 0 # simulation start time in seconds T_SIM_END = 24 * 60 * 60 # simulation end time in seconds -COM_STEP = 60 * 60 * 0.25 # 15 min communication step in seconds - +COM_STEP = 60 * 60 # 60 min communication step in seconds # ## Main script if __name__ == '__main__': @@ -64,14 +62,15 @@ # define a list for storing historical data history_weather_station = [] - # ToDo: create a MQTTv5 client with paho-mqtt + # ToDo: Create an MQTTv5 client with paho-mqtt. mqttc = ... # set user data if required mqttc.username_pw_set(username=MQTT_USER, password=MQTT_PW) + # ToDo: Define a callback function that will be executed when the client # receives message on a subscribed topic. It should decode your message - # and store the information for later in our history - # Note: do not change function's signature + # and store the information for later in our history. + # Note: Do not change the function's signature! def on_message(client, userdata, msg): """ Callback function for incoming messages @@ -79,7 +78,7 @@ def on_message(client, userdata, msg): # decode the payload payload = msg.payload.decode('utf-8') # ToDo: Parse the payload using the `json` package and write it to - # the history + # the history. ... pass @@ -88,7 +87,7 @@ def on_message(client, userdata, msg): # or a topic specific callback with `mqttc.message_callback_add()` mqttc.on_message = on_message - # ToDO: connect to the mqtt broker and subscribe to your topic + # ToDo: Connect to the mqtt broker and subscribe to your topic. mqtt_url = urlparse(MQTT_BROKER_URL) ... @@ -98,27 +97,27 @@ def on_message(client, userdata, msg): - # ToDo: print and subscribe to the weather station topic + # ToDo: Print and subscribe to the weather station topic. print(f"WeatherStation topic:\n {TOPIC_WEATHER_STATION}") mqttc.subscribe(topic=TOPIC_WEATHER_STATION) # create a non-blocking thread for mqtt communication mqttc.loop_start() - # ToDo: Create a loop that publishes every second a message to the broker + # ToDo: Create a loop that publishes every 0.2 seconds a message to the broker # that holds the simulation time "t_sim" and the corresponding temperature - # "t_amb" + # "t_amb". for t_sim in range(sim_model.t_start, int(sim_model.t_end + COM_STEP), int(COM_STEP)): - # ToDo: publish the simulated ambient temperature + # ToDo: Publish the simulated ambient temperature. ... # simulation step for next loop sim_model.do_step(int(t_sim + COM_STEP)) - time.sleep(1) + time.sleep(0.2) # close the mqtt listening thread mqttc.loop_stop() @@ -127,9 +126,9 @@ def on_message(client, userdata, msg): # plot results fig, ax = plt.subplots() - t_simulation = [item["t_sim"] for item in history_weather_station] + t_simulation = [item["t_sim"]/3600 for item in history_weather_station] temperature = [item["t_amb"] for item in history_weather_station] ax.plot(t_simulation, temperature) - ax.set_xlabel('time in s') + ax.set_xlabel('time in h') ax.set_ylabel('ambient temperature in °C') plt.show() diff --git a/tutorials/ngsi_v2/e1_virtual_weatherstation/e1_virtual_weatherstation_solution.py b/tutorials/ngsi_v2/e1_virtual_weatherstation/e1_virtual_weatherstation_solution.py index 842fd637..9e675ef0 100644 --- a/tutorials/ngsi_v2/e1_virtual_weatherstation/e1_virtual_weatherstation_solution.py +++ b/tutorials/ngsi_v2/e1_virtual_weatherstation/e1_virtual_weatherstation_solution.py @@ -3,7 +3,7 @@ # Create a virtual IoT device that simulates the ambient temperature and # publishes it via MQTT. The simulation function is already predefined. -# This exercise to give a simple introduction to the communication via MQTT. +# This exercise gives a simple introduction to the communication via MQTT. # The input sections are marked with 'ToDo' @@ -31,16 +31,15 @@ # import simulation model from tutorials.ngsi_v2.simulation_model import SimulationModel - # ## Parameters -# ToDo: Enter your mqtt broker url and port, e.g mqtt://test.mosquitto.org:1883 -MQTT_BROKER_URL = "mqtt://test.mosquitto.org:1883" -# ToDo: If required enter your username and password +# ToDo: Enter your mqtt broker url and port, e.g. mqtt://test.mosquitto.org:1883. +MQTT_BROKER_URL = "mqtt://test.mosquitto.org:1883" +# ToDo: If required, enter your username and password. MQTT_USER = "" -MQTT_PW = "" +MQTT_PW = "" # ToDo: Create a unique topic that your weather station will publish on, -# e.g. by using a uuid +# e.g. by using a uuid. UNIQUE_ID = str(uuid4()) TOPIC_WEATHER_STATION = f"fiware_workshop/{UNIQUE_ID}/weather_station" @@ -50,8 +49,7 @@ T_SIM_START = 0 # simulation start time in seconds T_SIM_END = 24 * 60 * 60 # simulation end time in seconds -COM_STEP = 60 * 60 * 1 # 60 min communication step in seconds - +COM_STEP = 60 * 60 # 60 min communication step in seconds # ## Main script if __name__ == '__main__': @@ -61,17 +59,19 @@ temp_max=TEMPERATURE_MAX, temp_min=TEMPERATURE_MIN) - # define lists to store historical data + # define a list for storing historical data history_weather_station = [] - # ToDo: create a MQTTv5 client with paho-mqtt - mqttc = mqtt.Client(protocol=mqtt.MQTTv5, callback_api_version=mqtt.CallbackAPIVersion.VERSION2) + # ToDo: Create an MQTTv5 client with paho-mqtt. + mqttc = mqtt.Client(protocol=mqtt.MQTTv5, + callback_api_version=mqtt.CallbackAPIVersion.VERSION2) # set user data if required mqttc.username_pw_set(username=MQTT_USER, password=MQTT_PW) + # ToDo: Define a callback function that will be executed when the client # receives message on a subscribed topic. It should decode your message - # and store the information for later in our history - # Note: do not change function's signature! + # and store the information for later in our history. + # Note: Do not change the function's signature! def on_message(client, userdata, msg): """ Callback function for incoming messages @@ -79,16 +79,15 @@ def on_message(client, userdata, msg): # decode the payload payload = msg.payload.decode('utf-8') # ToDo: Parse the payload using the `json` package and write it to - # the history + # the history. history_weather_station.append(json.loads(payload)) - # add your callback function to the client. You can either use a global # or a topic specific callback with `mqttc.message_callback_add()` mqttc.on_message = on_message - # ToDO: connect to the mqtt broker and subscribe to your topic + # ToDo: Connect to the mqtt broker and subscribe to your topic. mqtt_url = urlparse(MQTT_BROKER_URL) mqttc.connect(host=mqtt_url.hostname, port=mqtt_url.port, @@ -98,27 +97,27 @@ def on_message(client, userdata, msg): clean_start=mqtt.MQTT_CLEAN_START_FIRST_ONLY, properties=None) - # ToDo: print and subscribe to the weather station topic + # ToDo: Print and subscribe to the weather station topic. print(f"WeatherStation topic:\n {TOPIC_WEATHER_STATION}") mqttc.subscribe(topic=TOPIC_WEATHER_STATION) # create a non-blocking thread for mqtt communication mqttc.loop_start() - # ToDo: Create a loop that publishes every second a message to the broker + # ToDo: Create a loop that publishes every 0.2 seconds a message to the broker # that holds the simulation time "t_sim" and the corresponding temperature - # "t_amb" + # "t_amb". for t_sim in range(sim_model.t_start, int(sim_model.t_end + COM_STEP), int(COM_STEP)): - # ToDo: publish the simulated ambient temperature + # ToDo: Publish the simulated ambient temperature. mqttc.publish(topic=TOPIC_WEATHER_STATION, payload=json.dumps({"t_amb": sim_model.t_amb, "t_sim": sim_model.t_sim})) # simulation step for next loop sim_model.do_step(int(t_sim + COM_STEP)) - time.sleep(1) + time.sleep(0.2) # close the mqtt listening thread mqttc.loop_stop() @@ -127,9 +126,9 @@ def on_message(client, userdata, msg): # plot results fig, ax = plt.subplots() - t_simulation = [item["t_sim"] for item in history_weather_station] + t_simulation = [item["t_sim"]/3600 for item in history_weather_station] temperature = [item["t_amb"] for item in history_weather_station] ax.plot(t_simulation, temperature) - ax.set_xlabel('time in s') + ax.set_xlabel('time in h') ax.set_ylabel('ambient temperature in °C') plt.show() diff --git a/tutorials/ngsi_v2/e2_healthcheck/e2_healthcheck.py b/tutorials/ngsi_v2/e2_healthcheck/e2_healthcheck.py index 26b67e24..1ce7af44 100644 --- a/tutorials/ngsi_v2/e2_healthcheck/e2_healthcheck.py +++ b/tutorials/ngsi_v2/e2_healthcheck/e2_healthcheck.py @@ -1,14 +1,14 @@ """ # # Exercise 2: Service Health Check -# Create one or multiple filip clients and check if the corresponding services +# Create one or multiple FiLiP clients and check if the corresponding services # are up and running by accessing their version information. # The input sections are marked with 'ToDo' # #### Steps to complete: # 1. Set up the missing parameters in the parameter section -# 2. Create filip ngsi_v2 clients for the individual services and check for +# 2. Create FiLiP ngsi_v2 clients for the individual services and check for # their version # 3. Create a config object for the ngsi_v2 multi client (HttpClient), # create the multi client and again check for services' versions @@ -23,29 +23,32 @@ QuantumLeapClient # ## Parameters -# ToDo: Enter your context broker url and port, e.g. http://localhost:1026 +# ToDo: Enter your context broker url and port, e.g. http://localhost:1026. CB_URL = "http://localhost:1026" -# ToDo: Enter your IoT-Agent url and port, e.g. http://localhost:4041 +# ToDo: Enter your IoT-Agent url and port, e.g. http://localhost:4041. IOTA_URL = "http://localhost:4041" -# ToDo: Enter your QuantumLeap url and port, e.g. http://localhost:8668 +# ToDo: Enter your QuantumLeap url and port, e.g. http://localhost:8668. QL_URL = "http://localhost:8668" # ## Main script if __name__ == "__main__": - # Create a single client for each service and check the service for - # its version + # ToDo: Create a single client for each service and check the service for + # its version. cbc = ContextBrokerClient(url=CB_URL) - print(cbc.get_version()) + print(f"Context Broker Client: {cbc.get_version()}") iotac = ... + print(f"IoTA Client: {iotac.get_version()}") qlc = ... + print(f"Quantum Leap Client: {qlc.get_version()}") - # ToDo: Create a configuration object for a multi client + # ToDo: Create a configuration object for a multi client. config = HttpClientConfig(...) - # ToDo: Create a multi client check again all services for their version + # ToDo: Create a multi client check again all services for their version. multic = HttpClient(config=config) - print(multic.cb.get_version()) - ... + print(f"Multi Client (Context Broker): {multic.cb.get_version()}\n" + f"Multi Client (IoTA): {multic.iota.get_version()}\n" + f"Multi Client (Quantum Leap): {multic.timeseries.get_version()}") diff --git a/tutorials/ngsi_v2/e2_healthcheck/e2_healthcheck_solution.py b/tutorials/ngsi_v2/e2_healthcheck/e2_healthcheck_solution.py index f710ad5b..9cda3daf 100644 --- a/tutorials/ngsi_v2/e2_healthcheck/e2_healthcheck_solution.py +++ b/tutorials/ngsi_v2/e2_healthcheck/e2_healthcheck_solution.py @@ -1,14 +1,14 @@ """ # # Exercise 2: Service Health Check -# Create one or multiple filip clients and check if the corresponding services +# Create one or multiple FiLiP clients and check if the corresponding services # are up and running by accessing their version information. # The input sections are marked with 'ToDo' # #### Steps to complete: # 1. Set up the missing parameters in the parameter section -# 2. Create filip ngsi_v2 clients for the individual services and check for +# 2. Create FiLiP ngsi_v2 clients for the individual services and check for # their version # 3. Create a config object for the ngsi_v2 multi client (HttpClient), # create the multi client and again check for services' versions @@ -23,32 +23,32 @@ QuantumLeapClient # ## Parameters -# ToDo: Enter your context broker url and port, e.g. http://localhost:1026 +# ToDo: Enter your context broker url and port, e.g. http://localhost:1026. CB_URL = "http://localhost:1026" -# ToDo: Enter your IoT-Agent url and port, e.g. http://localhost:4041 +# ToDo: Enter your IoT-Agent url and port, e.g. http://localhost:4041. IOTA_URL = "http://localhost:4041" -# ToDo: Enter your QuantumLeap url and port, e.g. http://localhost:8668 +# ToDo: Enter your QuantumLeap url and port, e.g. http://localhost:8668. QL_URL = "http://localhost:8668" # ## Main script if __name__ == "__main__": # ToDo: Create a single client for each service and check the service for - # its version + # its version. cbc = ContextBrokerClient(url=CB_URL) - print(cbc.get_version()) + print(f"Context Broker Client: {cbc.get_version()}") iotac = IoTAClient(url=IOTA_URL) - print(iotac.get_version()) + print(f"IoTA Client: {iotac.get_version()}") qlc = QuantumLeapClient(url=QL_URL) - print(qlc.get_version()) + print(f"Quantum Leap Client: {qlc.get_version()}") - # ToDo: Create a configuration object for a multi client + # ToDo: Create a configuration object for a multi client. config = HttpClientConfig(cb_url=CB_URL, iota_url=IOTA_URL, ql_url=QL_URL) - # ToDo: Create a multi client check again all services for their version + # ToDo: Create a multi client check again all services for their version. multic = HttpClient(config=config) - print(multic.cb.get_version()) - print(multic.iota.get_version()) - print(multic.timeseries.get_version()) + print(f"Multi Client (Context Broker): {multic.cb.get_version()}\n" + f"Multi Client (IoTA): {multic.iota.get_version()}\n" + f"Multi Client (Quantum Leap): {multic.timeseries.get_version()}") diff --git a/tutorials/ngsi_v2/e3_context_entities/e3_context_entities.py b/tutorials/ngsi_v2/e3_context_entities/e3_context_entities.py index 80c38752..509f9316 100644 --- a/tutorials/ngsi_v2/e3_context_entities/e3_context_entities.py +++ b/tutorials/ngsi_v2/e3_context_entities/e3_context_entities.py @@ -1,11 +1,11 @@ """ # # Exercise 3: Context Entities and Relationships -# Create building context entity of type 'Building' according to FIWARE's +# Create a building context entity of type 'Building' according to FIWARE's # SmartData Models with the properties: `id`, `type`, `address`, `category`, # https://github.com/smart-data-models/dataModel.Building/blob/master/Building/doc/spec.md -# For the single properties check on the "Data Model description of +# For the single properties check the "Data Model description of # properties" section. The input sections are marked with 'ToDo' # #### Steps to complete: @@ -18,12 +18,12 @@ # ContextBroker. Afterwards, check if the Context Broker returns the # correct information about your building # 6. Create an `opening hours` attribute add them to the server -# 7. Retrieve the `opening hours` manipulate them and update the model at the +# 7. Retrieve the `opening hours`, manipulate them and update the model in the # server # 8. Repeat the procedure with a thermal zone. Currently, the smart data # models hold no definition of a thermal zone. Therefore, we first only add a # description attribute. -# 9. Add a `Relationship` attribute to your thermal zone with name +# 9. Add a `Relationship` attribute to your thermal zone with the name # `refBuilding` and type `Relationship` pointing to your building and post # the model to the context broker # 10. Add a `Relationship` attribute to your building with name @@ -44,21 +44,24 @@ from filip.utils.simple_ql import QueryString # ## Parameters -# ToDo: Enter your context broker host and port, e.g http://localhost:1026 +# ToDo: Enter your context broker host and port, e.g. http://localhost:1026. CB_URL = "http://localhost:1026" +# ToDo: Change the name of your service to something unique. If you run +# on a shared instance this is very important in order to avoid user +# collisions. You will use this service through the whole tutorial. +# If you forget to change it, an error will be raised! # FIWARE-Service SERVICE = 'filip_tutorial' -# FIWARE-Servicepath -# ToDo: Change the name of your service-path to something unique. If you run -# on a shared instance this very important in order to avoid user -# collisions. You will use this service path through the whole tutorial. -# If you forget to change it an error will be raised! -SERVICE_PATH = '/' +# FIWARE-Service path +SERVICE_PATH = '/' # ToDo: Path to json-files to store entity data for follow up exercises, -# e.g. ../e3_my_entities.json -WRITE_ENTITIES_FILEPATH = Path("") +# e.g. ../e3_my_entities.json. Files that are used in exercises and files +# that are used in solutions are different from each other so be careful +# when working with them. You can of course change the paths as you wish, +# but it is recommended to use the already given path names. +WRITE_ENTITIES_FILEPATH = Path("../e3_context_entities.json") # ## Main script if __name__ == '__main__': @@ -78,7 +81,7 @@ type="Array", value=["office"]) - # ToDo: create a property `address` for your building. Follow the full yaml + # ToDo: Create a property `address` for your building. Follow the full yaml # description in the specifications. It reuses the specification from # here: https://schema.org/PostalAddress address = NamedContextAttribute(name="address", @@ -89,7 +92,7 @@ - # ToDo: create a `description` property for your building + # ToDo: Create a `description` property for your building. building_description = NamedContextAttribute(...) @@ -103,10 +106,10 @@ category, address]) - # ToDo: Create a context broker client and add the fiware_header + # ToDo: Create a context broker client and add the fiware_header. cbc = ... # ToDo: Send your building model to the context broker. Check the client - # for the proper function + # for proper functioning. ... # Update your local building model with the one from the server @@ -114,9 +117,9 @@ entity_type=building.type) # print your `building model` as json - print(f"This is your building model: \n {building.json(indent=2)} \n") + print(f"This is your building model: \n {building.model_dump_json(indent=2)} \n") - # ToDo: create a `opening hours` property and add it to the building object + # ToDo: Create an `opening hours` property and add it to the building object # in the context broker. Do not update the whole entity! In real # scenarios it might have been modified by other users. opening_hours = NamedContextAttribute(name="openingHours", @@ -132,13 +135,13 @@ entity_type=building.type, attrs=[opening_hours]) - # ToDo: retrieve and print the opening hours + # ToDo: Retrieve and print the property `opening hours`. hours = cbc.get_attribute_value(...) - print(f"Your opening hours: {hours} \n" ) + print(f"Your opening hours: {hours} \n") - # ToDo: modify the `opening hours` of the building + # ToDo: Modify the property `opening hours` of the building. cbc.update_attribute_value(...) @@ -152,10 +155,10 @@ entity_type=building.type) # print your building - print(f"Your updated building model: \n {building.json(indent=2)}") + print(f"Your updated building model: \n {building.model_dump_json(indent=2)} \n") - # ToDo: Create an entity of thermal zone and add a description property - # to it + # ToDo: Create an entity of the thermal zone and add a description property + # to it. thermal_zone = ContextEntity(id="ThermalZone:001", type="ThermalZone") @@ -165,8 +168,8 @@ "the entire building") thermal_zone.add_attributes(attrs=[thermal_zone_description]) - # ToDo: Create and a property that references your building model. Use the - # `Relationship` for type and `refBuilding` for its name + # ToDo: Create and add a property that references your building model. Use the + # `Relationship` for type and `refBuilding` for its name. ref_building = NamedContextAttribute(name="refBuilding", type="Relationship", value=building.id) @@ -174,15 +177,15 @@ # print all relationships of your thermal zone for relationship in thermal_zone.get_relationships(): - print(f"Relationship properties of your thermal zone mdoel: \n " - f"{relationship.json(indent=2)} \n") + print(f"Relationship properties of your thermal zone model: \n " + f"{relationship.model_dump_json(indent=2)} \n") - # ToDo: Post your thermal zone model to the context broker + # ToDo: Post your thermal zone model to the context broker. ... ... - # ToDo: Create and a property that references your thermal zone. Use the + # ToDo: Create and add a property that references your thermal zone. Use the # `Relationship` for type and `hasZone` for its name. Make sure that # your local model and the server model are in sync afterwards. ref_zone = NamedContextAttribute(...) @@ -194,34 +197,34 @@ building = cbc.get_entity(entity_id=building.id, entity_type=building.type) - # ToDo: create a filter request that retrieves all entities from the - # server`that have `refBuilding` attribute that references your building + # ToDo: Create a filter request that retrieves all entities from the + # server that have `refBuilding` attribute that reference your building # by using the FIWARE's simple query language. - # `filip.utils.simple_ql` module helps you to validate your query string - # 1. prepare the query string using the `filip.utils.simple_ql` - # 2. use the string in a context broker request and retrieve the entities. + # `filip.utils.simple_ql` module helps you to validate your query string. + # 1. Prepare the query string using the `filip.utils.simple_ql`. + # 2. Use the string in a context broker request and retrieve the entities. query = QueryString(qs=("refBuilding", "==", building.id)) for entity in cbc.get_entity_list(q=query): print(f"All entities referencing the building: " - f"\n {entity.json(indent=2)}\n") + f"\n {entity.model_dump_json(indent=2)}\n") - # ToDo: create a filter request that retrieves all entities from the - # server`that have `hasZone' attribute that references your thermal zone + # ToDo: Create a filter request that retrieves all entities from the + # server that have `hasZone` attribute that reference your thermal zone # by using the FIWARE's simple query language. - # `filip.utils.simple_ql` module helps you to validate your query string - # 1. prepare the query string using the `filip.utils.simple_ql` - # 2. use the string in a context broker request and retrieve the entities. + # `filip.utils.simple_ql` module helps you to validate your query string. + # 1. Prepare the query string using the `filip.utils.simple_ql`. + # 2. Use the string in a context broker request and retrieve the entities. query = ... for entity in cbc.get_entity_list(q=query): print(f"All entities referencing the thermal zone: " - f"\n {entity.json(indent=2)} \n") + f"\n {entity.model_dump_json(indent=2)} \n") # write entities to file and clear server state assert WRITE_ENTITIES_FILEPATH.suffix == '.json', \ f"Wrong file extension! {WRITE_ENTITIES_FILEPATH.suffix}" WRITE_ENTITIES_FILEPATH.touch(exist_ok=True) with WRITE_ENTITIES_FILEPATH.open('w', encoding='utf-8') as f: - entities = [item.dict() for item in cbc.get_entity_list()] + entities = [item.model_dump() for item in cbc.get_entity_list()] json.dump(entities, f, ensure_ascii=False, indent=2) clear_context_broker(url=CB_URL, fiware_header=fiware_header) diff --git a/tutorials/ngsi_v2/e3_context_entities/e3_context_entities_solution.py b/tutorials/ngsi_v2/e3_context_entities/e3_context_entities_solution.py index a3cb0483..69cb7d1f 100644 --- a/tutorials/ngsi_v2/e3_context_entities/e3_context_entities_solution.py +++ b/tutorials/ngsi_v2/e3_context_entities/e3_context_entities_solution.py @@ -1,11 +1,11 @@ """ # # Exercise 3: Context Entities and Relationships -# Create building context entity of type 'Building' according to FIWARE's +# Create a building context entity of type 'Building' according to FIWARE's # SmartData Models with the properties: `id`, `type`, `address`, `category`, # https://github.com/smart-data-models/dataModel.Building/blob/master/Building/doc/spec.md -# For the single properties check on the "Data Model description of +# For the single properties check the "Data Model description of # properties" section. The input sections are marked with 'ToDo' # #### Steps to complete: @@ -18,12 +18,12 @@ # ContextBroker. Afterwards, check if the Context Broker returns the # correct information about your building # 6. Create an `opening hours` attribute add them to the server -# 7. Retrieve the `opening hours` manipulate them and update the model at the +# 7. Retrieve the `opening hours`, manipulate them and update the model in the # server # 8. Repeat the procedure with a thermal zone. Currently, the smart data # models hold no definition of a thermal zone. Therefore, we first only add a # description attribute. -# 9. Add a `Relationship` attribute to your thermal zone with name +# 9. Add a `Relationship` attribute to your thermal zone with the name # `refBuilding` and type `Relationship` pointing to your building and post # the model to the context broker # 10. Add a `Relationship` attribute to your building with name @@ -44,20 +44,23 @@ from filip.utils.simple_ql import QueryString # ## Parameters -# ToDo: Enter your context broker host and port, e.g http://localhost:1026 +# ToDo: Enter your context broker host and port, e.g. http://localhost:1026. CB_URL = "http://localhost:1026" +# ToDo: Change the name of your service to something unique. If you run +# on a shared instance this is very important in order to avoid user +# collisions. You will use this service through the whole tutorial. +# If you forget to change it, an error will be raised! # FIWARE-Service SERVICE = 'filip_tutorial' -# FIWARE-Servicepath -# ToDo: Change the name of your service-path to something unique. If you run -# on a shared instance this very important in order to avoid user -# collisions. You will use this service path through the whole tutorial. -# If you forget to change it an error will be raised! -SERVICE_PATH = '/your_path' +# FIWARE-Service path +SERVICE_PATH = '/' # ToDo: Path to json-files to store entity data for follow up exercises, -# e.g. ./e3_my_entities.json +# e.g. ../e3_my_entities.json. Files that are used in exercises and files +# that are used in solutions are different from each other so be careful +# when working with them. You can of course change the paths as you wish, +# but it is recommended to use the already given path names. WRITE_ENTITIES_FILEPATH = Path("../e3_context_entities_solution_entities.json") # ## Main script @@ -78,7 +81,7 @@ type="Array", value=["office"]) - # ToDo: create a property `address` for your building. Follow the full yaml + # ToDo: Create a property `address` for your building. Follow the full yaml # description in the specifications. It reuses the specification from # here: https://schema.org/PostalAddress address = NamedContextAttribute(name="address", @@ -90,7 +93,7 @@ "streetAddress": "Any Street 5" }) - # ToDo: create a `description` property for your building + # ToDo: Create a `description` property for your building. building_description = NamedContextAttribute(name="description", type="Text", value="Small office building " @@ -103,10 +106,10 @@ category, address]) - # ToDo: Create a context broker client and add the fiware_header + # ToDo: Create a context broker client and add the fiware_header. cbc = ContextBrokerClient(url=CB_URL, fiware_header=fiware_header) # ToDo: Send your building model to the context broker. Check the client - # for the proper function + # for proper functioning. cbc.post_entity(entity=building) # Update your local building model with the one from the server @@ -114,9 +117,9 @@ entity_type=building.type) # print your `building model` as json - print(f"This is your building model: \n {building.json(indent=2)} \n") + print(f"This is your building model: \n {building.model_dump_json(indent=2)} \n") - # ToDo: create a `opening hours` property and add it to the building object + # ToDo: Create an `opening hours` property and add it to the building object # in the context broker. Do not update the whole entity! In real # scenarios it might have been modified by other users. opening_hours = NamedContextAttribute(name="openingHours", @@ -132,13 +135,13 @@ entity_type=building.type, attrs=[opening_hours]) - # ToDo: retrieve and print the opening hours + # ToDo: Retrieve and print the property `opening hours`. hours = cbc.get_attribute_value(entity_id=building.id, entity_type=building.type, attr_name=opening_hours.name) - print(f"Your opening hours: {hours} \n" ) + print(f"Your opening hours: {hours} \n") - # ToDo: modify the `opening hours` of the building + # ToDo: Modify the property `opening hours` of the building. cbc.update_attribute_value(entity_id=building.id, entity_type=building.type, attr_name=opening_hours.name, @@ -152,10 +155,10 @@ entity_type=building.type) # print your building - print(f"Your updated building model: \n {building.json(indent=2)}") + print(f"Your updated building model: \n {building.model_dump_json(indent=2)} \n") - # ToDo: Create an entity of thermal zone and add a description property - # to it + # ToDo: Create an entity of the thermal zone and add a description property + # to it. thermal_zone = ContextEntity(id="ThermalZone:001", type="ThermalZone") @@ -165,8 +168,8 @@ "the entire building") thermal_zone.add_attributes(attrs=[thermal_zone_description]) - # ToDo: Create and a property that references your building model. Use the - # `Relationship` for type and `refBuilding` for its name + # ToDo: Create and add a property that references your building model. Use the + # `Relationship` for type and `refBuilding` for its name. ref_building = NamedContextAttribute(name="refBuilding", type="Relationship", value=building.id) @@ -174,15 +177,15 @@ # print all relationships of your thermal zone for relationship in thermal_zone.get_relationships(): - print(f"Relationship properties of your thermal zone mdoel: \n " - f"{relationship.json(indent=2)} \n") + print(f"Relationship properties of your thermal zone model: \n " + f"{relationship.model_dump_json(indent=2)} \n") - # ToDo: Post your thermal zone model to the context broker + # ToDo: Post your thermal zone model to the context broker. cbc.post_entity(entity=thermal_zone) thermal_zone = cbc.get_entity(entity_id=thermal_zone.id, entity_type=thermal_zone.type) - # ToDo: Create and a property that references your thermal zone. Use the + # ToDo: Create and add a property that references your thermal zone. Use the # `Relationship` for type and `hasZone` for its name. Make sure that # your local model and the server model are in sync afterwards. ref_zone = NamedContextAttribute(name="hasZone", @@ -194,34 +197,34 @@ building = cbc.get_entity(entity_id=building.id, entity_type=building.type) - # ToDo: create a filter request that retrieves all entities from the - # server`that have `refBuilding` attribute that references your building + # ToDo: Create a filter request that retrieves all entities from the + # server that have `refBuilding` attribute that reference your building # by using the FIWARE's simple query language. - # `filip.utils.simple_ql` module helps you to validate your query string - # 1. prepare the query string using the `filip.utils.simple_ql` - # 2. use the string in a context broker request and retrieve the entities. + # `filip.utils.simple_ql` module helps you to validate your query string. + # 1. Prepare the query string using the `filip.utils.simple_ql`. + # 2. Use the string in a context broker request and retrieve the entities. query = QueryString(qs=("refBuilding", "==", building.id)) for entity in cbc.get_entity_list(q=query): print(f"All entities referencing the building: " - f"\n {entity.json(indent=2)}\n") + f"\n {entity.model_dump_json(indent=2)}\n") - # ToDo: create a filter request that retrieves all entities from the - # server`that have `hasZone' attribute that references your thermal zone + # ToDo: Create a filter request that retrieves all entities from the + # server that have `hasZone` attribute that reference your thermal zone # by using the FIWARE's simple query language. - # `filip.utils.simple_ql` module helps you to validate your query string - # 1. prepare the query string using the `filip.utils.simple_ql` - # 2. use the string in a context broker request and retrieve the entities. + # `filip.utils.simple_ql` module helps you to validate your query string. + # 1. Prepare the query string using the `filip.utils.simple_ql`. + # 2. Use the string in a context broker request and retrieve the entities. query = QueryString(qs=("hasZone", "==", thermal_zone.id)) for entity in cbc.get_entity_list(q=query): print(f"All entities referencing the thermal zone: " - f"\n {entity.json(indent=2)} \n") + f"\n {entity.model_dump_json(indent=2)} \n") # write entities to file and clear server state assert WRITE_ENTITIES_FILEPATH.suffix == '.json', \ f"Wrong file extension! {WRITE_ENTITIES_FILEPATH.suffix}" WRITE_ENTITIES_FILEPATH.touch(exist_ok=True) with WRITE_ENTITIES_FILEPATH.open('w', encoding='utf-8') as f: - entities = [item.dict() for item in cbc.get_entity_list()] + entities = [item.model_dump() for item in cbc.get_entity_list()] json.dump(entities, f, ensure_ascii=False, indent=2) clear_context_broker(url=CB_URL, fiware_header=fiware_header) diff --git a/tutorials/ngsi_v2/e4_iot_thermal_zone_sensors/e4_iot_thermal_zone_sensors.py b/tutorials/ngsi_v2/e4_iot_thermal_zone_sensors/e4_iot_thermal_zone_sensors.py index efffa403..a1f80c7a 100644 --- a/tutorials/ngsi_v2/e4_iot_thermal_zone_sensors/e4_iot_thermal_zone_sensors.py +++ b/tutorials/ngsi_v2/e4_iot_thermal_zone_sensors/e4_iot_thermal_zone_sensors.py @@ -1,9 +1,9 @@ """ # # Exercise 4: Virtual Thermal Zone -# Create two virtual IoT device. One of them represents the temperature -# sensor for the air temperature of a the thermal zone, whereas the second -# represents a virtual weather station. Both devices publish there values to +# Create two virtual IoT devices. One of them represents the temperature +# sensor for the air temperature of a thermal zone, whereas the second +# represents a virtual weather station. Both devices publish their values to # the platform via MQTT. Use the simulation model of # e1_virtual_weatherstation.py # @@ -11,7 +11,7 @@ # # #### Steps to complete: # 1. Set up the missing parameters in the parameter section -# 2. Create a service group and two devices +# 2. Create a service group and two corresponding devices # 3. Provision the service group and the devices # 4. Create an MQTT client using the filip.client.mqtt package and register # your service group and your devices @@ -31,7 +31,7 @@ import paho.mqtt.client as mqtt # import from filip -from filip.clients.ngsi_v2 import ContextBrokerClient +from filip.clients.ngsi_v2 import ContextBrokerClient, IoTAClient from filip.clients.mqtt import IoTAMQTTClient from filip.models.base import FiwareHeader from filip.models.ngsi_v2.iot import Device, DeviceAttribute, ServiceGroup @@ -40,26 +40,31 @@ from tutorials.ngsi_v2.simulation_model import SimulationModel # ## Parameters -# ToDo: Enter your context broker host and port, e.g. http://localhost:1026 +# ToDo: Enter your context broker host and port, e.g. http://localhost:1026. CB_URL = "http://localhost:1026" -# ToDo: Enter your IoT-Agent host and port, e.g. http://localhost:4041 +# ToDo: Enter your IoT-Agent host and port, e.g. http://localhost:4041. IOTA_URL = "http://localhost:4041" -# ToDo: Enter your mqtt broker url, e.g mqtt://test.mosquitto.org:1883 +# ToDo: Enter your mqtt broker url, e.g. mqtt://test.mosquitto.org:1883. MQTT_BROKER_URL = "mqtt://localhost:1883" -# ToDo: If required enter your username and password +# ToDo: If required, enter your username and password. MQTT_USER = "" -MQTT_PW = "" +MQTT_PW = "" + +# ToDo: Change the name of your service to something unique. If you run +# on a shared instance this is very important in order to avoid user +# collisions. You will use this service through the whole tutorial. +# If you forget to change it, an error will be raised! # FIWARE-Service SERVICE = 'filip_tutorial' -# FIWARE-Servicepath -# ToDo: Change the name of your service-path to something unique. If you run -# on a shared instance this very important in order to avoid user -# collisions. You will use this service path through the whole tutorial. -# If you forget to change it an error will be raised! -SERVICE_PATH = '/' -APIKEY = SERVICE_PATH.strip('/') - -# Path to json-files to device configuration data for follow up exercises +# FIWARE-Service path +SERVICE_PATH = '/' + +# ToDo: Change the APIKEY to something unique. This represents the "token" +# for IoT devices to connect (send/receive data) with the platform. In the +# context of MQTT, APIKEY is linked with the topic used for communication. +APIKEY = 'your_apikey' + +# path to json-files to device configuration data for follow-up exercises WRITE_GROUPS_FILEPATH = Path("../e4_iot_thermal_zone_sensors_groups.json") WRITE_DEVICES_FILEPATH = Path("../e4_iot_thermal_zone_sensors_devices.json") @@ -78,8 +83,8 @@ fiware_header = FiwareHeader(service=SERVICE, service_path=SERVICE_PATH) # clear the state of your service and scope - clear_context_broker(url=CB_URL, fiware_header=fiware_header) clear_iot_agent(url=IOTA_URL, fiware_header=fiware_header) + clear_context_broker(url=CB_URL, fiware_header=fiware_header) # instantiate simulation model sim_model = SimulationModel(t_start=T_SIM_START, @@ -92,17 +97,16 @@ history_weather_station = [] history_zone_temperature_sensor = [] - # Create a service group and add it to your + # create a service group with your api key service_group = ServiceGroup(apikey=APIKEY, resource="/iot/json") - # ToDo: create two IoTA-MQTT devices for the weather station and the zone + # ToDo: Create two IoTA-MQTT devices for the weather station and the zone # temperature sensor. Also add the simulation time as `active attribute` # to each device! - # # create the weather station device - # create the simtime attribute and add during device creation - t_sim = DeviceAttribute(name='simtime', + # create the `sim_time` attribute and add it to the weather station's attributes + t_sim = DeviceAttribute(name='sim_time', object_id='t_sim', type="Number") @@ -116,7 +120,7 @@ commands=[]) # create a temperature attribute and add it via the api of the - # `device`-model. Use the 't_amb' as `object_id`. `object_id` specifies + # `device`-model. Use the `t_amb` as `object_id`. `object_id` specifies # what key will be used in the MQTT Message payload t_amb = DeviceAttribute(name='temperature', object_id='t_amb', @@ -124,8 +128,8 @@ weather_station.add_attribute(t_amb) - # ToDo: create the zone temperature device add the t_sim attribute upon - # creation + # ToDo: Create the zone temperature device and add the `t_sim` attribute upon + # creation. zone_temperature_sensor = Device(...) @@ -134,51 +138,50 @@ - # ToDo: Create the temperature attribute. Use the 't_zone' as `object_id`. - # `object_id` specifies what key will be used in the MQTT Message payload + # ToDo: Create the temperature attribute. Use the `t_zone` as `object_id`. + # `object_id` specifies what key will be used in the MQTT Message payload. t_zone = DeviceAttribute(...) - zone_temperature_sensor.add_attribute(t_zone) - # ToDo: Create an IoTAClient + # ToDo: Create an IoTAClient. iotac = ... - # ToDo: Provision service group and add it to your IoTAMQTTClient + # ToDo: Provision service group and add it to your IoTAMQTTClient. ... - # ToDo: Provision the devices at the IoTA-Agent - # provision the WeatherStation device + # ToDo: Provision the devices at the IoTA-Agent. + # provision the weather station device iotac.post_device(device=weather_station, update=True) - # ToDo: provision the zone temperature device + # ToDo: Provision the zone temperature device. ... - # ToDo: Check in the context broker if the entities corresponding to your - # devices where correctly created - # ToDo: Create a context broker client + # ToDo: Create a context broker client. + # ToDo: Check in the context broker whether the entities corresponding to your + # devices were correctly created. cbc = ContextBrokerClient(url=CB_URL, fiware_header=fiware_header) - # Get WeatherStation entity - print(cbc.get_entity(weather_station.entity_name).json(indent=2)) - # ToDo: Get ZoneTemperatureSensor entity + # get weather station entity + print(f"Weather station:\n{cbc.get_entity(weather_station.entity_name).model_dump_json(indent=2)}") + # ToDo: Get zone temperature sensor entity. print(...) - # ToDo: create an MQTTv5 client using filip.clients.mqtt.IoTAMQTTClient + # ToDo: Create an MQTTv5 client using filip.clients.mqtt.IoTAMQTTClient. mqttc = IoTAMQTTClient(protocol=...) - # ToDo: Register the service group with your MQTT-Client + # ToDo: Register the service group with your MQTT-Client. mqttc.add_service_group(service_group=service_group) - # ToDo: Register devices with your MQTT-Client + # ToDo: Register devices with your MQTT-Client. # register the weather station mqttc.add_device(weather_station) - # ToDo: register the zone temperature sensor + # ToDo: Register the zone temperature sensor. ... - # The IoTAMQTTClient automatically creates the outgoing topics from the + # The IoTAMQTTClient automatically creates outgoing topics from the # device configuration during runtime. Hence, we need to construct them - # manually in order to subscribe to them. This is usually not required as - # only the platform should listen to incoming traffic. - # if you want to listen subscribe to the following topics: + # manually in order to subscribe to them. This is usually not required as + # only the platform should listen to the incoming traffic. + # If you want to listen subscribe to the following topics: # "/json///attrs" # "/json///attrs" - # ToDO: connect to the mqtt broker and subscribe to your topic + # ToDO: Connect to the MQTT broker and subscribe to your topic. ... @@ -193,39 +196,39 @@ # create a non-blocking thread for mqtt communication mqttc.loop_start() - # ToDo: Create a loop that publishes every second a message to the broker - # that holds the simulation time "simtime" and the corresponding - # temperature "temperature" the loop should. You may use the `object_id` - # or the attribute name as key in your payload. + # ToDo: Create a loop that publishes a message every 100 milliseconds + # to the broker that holds the simulation time `sim_time` and the + # corresponding temperature `temperature`. You may use the `object_id` + # or the attribute name as a key in your payload. for t_sim in range(sim_model.t_start, sim_model.t_end + int(COM_STEP), int(COM_STEP)): # publish the simulated ambient temperature mqttc.publish(device_id=weather_station.device_id, payload={"temperature": sim_model.t_amb, - "simtime": sim_model.t_sim}) + "sim_time": sim_model.t_sim}) - # ToDo: publish the simulated zone temperature + # ToDo: Publish the simulated zone temperature. ... - # simulation step for next loop + # simulation step for the next loop sim_model.do_step(int(t_sim + COM_STEP)) # wait for one second before publishing the next values - time.sleep(1) + time.sleep(0.1) - # Get corresponding entities and write values to history + # get corresponding entities and store the data weather_station_entity = cbc.get_entity( entity_id=weather_station.entity_name, entity_type=weather_station.entity_type ) # append the data to the local history history_weather_station.append( - {"simtime": weather_station_entity.simtime.value, + {"sim_time": weather_station_entity.sim_time.value, "temperature": weather_station_entity.temperature.value}) - # ToDo: Get ZoneTemperatureSensor and write values to history + # ToDo: Get zone temperature sensor and store the data. zone_temperature_sensor_entity = ... @@ -240,20 +243,22 @@ # disconnect the mqtt device mqttc.disconnect() - # plot results + # plot the results fig, ax = plt.subplots() - t_simulation = [item["simtime"] for item in history_weather_station] + t_simulation = [item["sim_time"]/3600 for item in history_weather_station] temperature = [item["temperature"] for item in history_weather_station] ax.plot(t_simulation, temperature) - ax.set_xlabel('time in s') + ax.title.set_text("Weather Station") + ax.set_xlabel('time in h') ax.set_ylabel('ambient temperature in °C') fig2, ax2 = plt.subplots() - t_simulation = [item["simtime"] for item in history_zone_temperature_sensor] + t_simulation = [item["sim_time"]/3600 for item in history_zone_temperature_sensor] temperature = [item["temperature"] for item in history_zone_temperature_sensor] ax2.plot(t_simulation, temperature) - ax2.set_xlabel('time in s') + ax2.title.set_text("Zone Temperature Sensor") + ax2.set_xlabel('time in h') ax2.set_ylabel('zone temperature in °C') plt.show() diff --git a/tutorials/ngsi_v2/e4_iot_thermal_zone_sensors/e4_iot_thermal_zone_sensors_solution.py b/tutorials/ngsi_v2/e4_iot_thermal_zone_sensors/e4_iot_thermal_zone_sensors_solution.py index e16d82dd..74e7de09 100644 --- a/tutorials/ngsi_v2/e4_iot_thermal_zone_sensors/e4_iot_thermal_zone_sensors_solution.py +++ b/tutorials/ngsi_v2/e4_iot_thermal_zone_sensors/e4_iot_thermal_zone_sensors_solution.py @@ -1,9 +1,9 @@ """ # # Exercise 4: Virtual Thermal Zone -# Create two virtual IoT device. One of them represents the temperature -# sensor for the air temperature of a the thermal zone, whereas the second -# represents a virtual weather station. Both devices publish there values to +# Create two virtual IoT devices. One of them represents the temperature +# sensor for the air temperature of a thermal zone, whereas the second +# represents a virtual weather station. Both devices publish their values to # the platform via MQTT. Use the simulation model of # e1_virtual_weatherstation.py # @@ -11,7 +11,7 @@ # # #### Steps to complete: # 1. Set up the missing parameters in the parameter section -# 2. Create a service group and two devices +# 2. Create a service group and two corresponding devices # 3. Provision the service group and the devices # 4. Create an MQTT client using the filip.client.mqtt package and register # your service group and your devices @@ -40,30 +40,33 @@ from tutorials.ngsi_v2.simulation_model import SimulationModel # ## Parameters -# ToDo: Enter your context broker host and port, e.g. http://localhost:1026 +# ToDo: Enter your context broker host and port, e.g. http://localhost:1026. CB_URL = "http://localhost:1026" -# ToDo: Enter your IoT-Agent host and port, e.g. http://localhost:4041 +# ToDo: Enter your IoT-Agent host and port, e.g. http://localhost:4041. IOTA_URL = "http://localhost:4041" -# ToDo: Enter your mqtt broker url, e.g mqtt://test.mosquitto.org:1883 +# ToDo: Enter your mqtt broker url, e.g. mqtt://test.mosquitto.org:1883. MQTT_BROKER_URL = "mqtt://localhost:1883" -# ToDo: If required enter your username and password +# ToDo: If required, enter your username and password. MQTT_USER = "" -MQTT_PW = "" +MQTT_PW = "" + +# ToDo: Change the name of your service to something unique. If you run +# on a shared instance this is very important in order to avoid user +# collisions. You will use this service through the whole tutorial. +# If you forget to change it, an error will be raised! # FIWARE-Service SERVICE = 'filip_tutorial' -# FIWARE-Servicepath -# ToDo: Change the name of your service-path to something unique. If you run -# on a shared instance this very important in order to avoid user -# collisions. You will use this service path through the whole tutorial. -# If you forget to change it an error will be raised! -SERVICE_PATH = '/your_path' -APIKEY = SERVICE_PATH.strip('/') - -# Path to json-files to device configuration data for follow up exercises -WRITE_GROUPS_FILEPATH = Path( - "../e4_iot_thermal_zone_sensors_solution_groups.json") -WRITE_DEVICES_FILEPATH = Path( - "../e4_iot_thermal_zone_sensors_solution_devices.json") +# FIWARE-Service path +SERVICE_PATH = '/' + +# ToDo: Change the APIKEY to something unique. This represents the "token" +# for IoT devices to connect (send/receive data) with the platform. In the +# context of MQTT, APIKEY is linked with the topic used for communication. +APIKEY = 'your_apikey' + +# path to json-files to device configuration data for follow-up exercises +WRITE_GROUPS_FILEPATH = Path("../e4_iot_thermal_zone_sensors_solution_groups.json") +WRITE_DEVICES_FILEPATH = Path("../e4_iot_thermal_zone_sensors_solution_devices.json") # set parameters for the temperature simulation TEMPERATURE_MAX = 10 # maximal ambient temperature @@ -80,8 +83,8 @@ fiware_header = FiwareHeader(service=SERVICE, service_path=SERVICE_PATH) # clear the state of your service and scope - clear_context_broker(url=CB_URL, fiware_header=fiware_header) clear_iot_agent(url=IOTA_URL, fiware_header=fiware_header) + clear_context_broker(url=CB_URL, fiware_header=fiware_header) # instantiate simulation model sim_model = SimulationModel(t_start=T_SIM_START, @@ -94,17 +97,16 @@ history_weather_station = [] history_zone_temperature_sensor = [] - # Create a service group and add it to your + # create a service group with your api key service_group = ServiceGroup(apikey=APIKEY, resource="/iot/json") - # ToDo: create two IoTA-MQTT devices for the weather station and the zone + # ToDo: Create two IoTA-MQTT devices for the weather station and the zone # temperature sensor. Also add the simulation time as `active attribute` # to each device! - # # create the weather station device - # create the simtime attribute and add during device creation - t_sim = DeviceAttribute(name='simtime', + # create the `sim_time` attribute and add it to the weather station's attributes + t_sim = DeviceAttribute(name='sim_time', object_id='t_sim', type="Number") @@ -118,7 +120,7 @@ commands=[]) # create a temperature attribute and add it via the api of the - # `device`-model. Use the 't_amb' as `object_id`. `object_id` specifies + # `device`-model. Use the `t_amb` as `object_id`. `object_id` specifies # what key will be used in the MQTT Message payload t_amb = DeviceAttribute(name='temperature', object_id='t_amb', @@ -126,8 +128,8 @@ weather_station.add_attribute(t_amb) - # ToDo: create the zone temperature device add the t_sim attribute upon - # creation + # ToDo: Create the zone temperature device and add the `t_sim` attribute upon + # creation. zone_temperature_sensor = Device(device_id='device:002', entity_name='urn:ngsi-ld:TemperatureSensor:001', entity_type='TemperatureSensor', @@ -136,53 +138,55 @@ apikey=APIKEY, attributes=[t_sim], commands=[]) - # ToDo: Create the temperature attribute. Use the 't_zone' as `object_id`. - # `object_id` specifies what key will be used in the MQTT Message payload + + # ToDo: Create the temperature attribute. Use the `t_zone` as `object_id`. + # `object_id` specifies what key will be used in the MQTT Message payload. t_zone = DeviceAttribute(name='temperature', object_id='t_zone', type="Number") + zone_temperature_sensor.add_attribute(t_zone) - # ToDo: Create an IoTAClient + # ToDo: Create an IoTAClient. iotac = IoTAClient(url=IOTA_URL, fiware_header=fiware_header) - # ToDo: Provision service group and add it to your IoTAMQTTClient + # ToDo: Provision service group and add it to your IoTAMQTTClient. iotac.post_group(service_group=service_group, update=True) - # ToDo: Provision the devices at the IoTA-Agent - # provision the WeatherStation device + # ToDo: Provision the devices at the IoTA-Agent. + # provision the weather station device iotac.post_device(device=weather_station, update=True) - # ToDo: provision the zone temperature device + # ToDo: Provision the zone temperature device. iotac.post_device(device=zone_temperature_sensor, update=True) - # ToDo: Check in the context broker if the entities corresponding to your - # devices where correctly created - # ToDo: Create a context broker client + # ToDo: Create a context broker client. + # ToDo: Check in the context broker whether the entities corresponding to your + # devices were correctly created. cbc = ContextBrokerClient(url=CB_URL, fiware_header=fiware_header) - # Get WeatherStation entity - print(cbc.get_entity(weather_station.entity_name).json(indent=2)) - # Get ZoneTemperatureSensor entity - print(cbc.get_entity(zone_temperature_sensor.entity_name).json(indent=2)) + # get weather station entity + print(f"Weather station:\n{cbc.get_entity(weather_station.entity_name).model_dump_json(indent=2)}") + # ToDo: Get zone temperature sensor entity. + print(f"Zone temperature sensor:\n{cbc.get_entity(zone_temperature_sensor.entity_name).model_dump_json(indent=2)}") - # ToDo: create an MQTTv5 client using filip.clients.mqtt.IoTAMQTTClient + # ToDo: Create an MQTTv5 client using filip.clients.mqtt.IoTAMQTTClient. mqttc = IoTAMQTTClient(protocol=mqtt.MQTTv5) # set user data if required mqttc.username_pw_set(username=MQTT_USER, password=MQTT_PW) - # ToDo: Register the service group with your MQTT-Client + # ToDo: Register the service group with your MQTT-Client. mqttc.add_service_group(service_group=service_group) - # ToDo: Register devices with your MQTT-Client + # ToDo: Register devices with your MQTT-Client. # register the weather station mqttc.add_device(weather_station) - # ToDo: register the zone temperature sensor + # ToDo: Register the zone temperature sensor. mqttc.add_device(zone_temperature_sensor) - # The IoTAMQTTClient automatically creates the outgoing topics from the + # The IoTAMQTTClient automatically creates outgoing topics from the # device configuration during runtime. Hence, we need to construct them - # manually in order to subscribe to them. This is usually not required as - # only the platform should listen to incoming traffic. - # if you want to listen subscribe to the following topics: + # manually in order to subscribe to them. This is usually not required as + # only the platform should listen to the incoming traffic. + # If you want to listen subscribe to the following topics: # "/json///attrs" # "/json///attrs" - # ToDO: connect to the mqtt broker and subscribe to your topic + # ToDO: Connect to the MQTT broker and subscribe to your topic. mqtt_url = urlparse(MQTT_BROKER_URL) mqttc.connect(host=mqtt_url.hostname, port=mqtt_url.port, @@ -197,45 +201,45 @@ # create a non-blocking thread for mqtt communication mqttc.loop_start() - # ToDo: Create a loop that publishes every second a message to the broker - # that holds the simulation time "simtime" and the corresponding - # temperature "temperature" the loop should. You may use the `object_id` - # or the attribute name as key in your payload. + # ToDo: Create a loop that publishes a message every 100 milliseconds + # to the broker that holds the simulation time `sim_time` and the + # corresponding temperature `temperature`. You may use the `object_id` + # or the attribute name as a key in your payload. for t_sim in range(sim_model.t_start, sim_model.t_end + int(COM_STEP), int(COM_STEP)): # publish the simulated ambient temperature mqttc.publish(device_id=weather_station.device_id, payload={"temperature": sim_model.t_amb, - "simtime": sim_model.t_sim}) + "sim_time": sim_model.t_sim}) - # ToDo: publish the simulated zone temperature + # ToDo: Publish the simulated zone temperature. mqttc.publish(device_id=zone_temperature_sensor.device_id, payload={"temperature": sim_model.t_zone, - "simtime": sim_model.t_sim}) + "sim_time": sim_model.t_sim}) - # simulation step for next loop + # simulation step for the next loop sim_model.do_step(int(t_sim + COM_STEP)) # wait for one second before publishing the next values - time.sleep(1) + time.sleep(0.1) - # Get corresponding entities and write values to history + # get corresponding entities and store the data weather_station_entity = cbc.get_entity( entity_id=weather_station.entity_name, entity_type=weather_station.entity_type ) # append the data to the local history history_weather_station.append( - {"simtime": weather_station_entity.simtime.value, + {"sim_time": weather_station_entity.sim_time.value, "temperature": weather_station_entity.temperature.value}) - # ToDo: Get ZoneTemperatureSensor and write values to history + # ToDo: Get zone temperature sensor and store the data. zone_temperature_sensor_entity = cbc.get_entity( entity_id=zone_temperature_sensor.entity_name, entity_type=zone_temperature_sensor.entity_type ) history_zone_temperature_sensor.append( - {"simtime": zone_temperature_sensor_entity.simtime.value, + {"sim_time": zone_temperature_sensor_entity.sim_time.value, "temperature": zone_temperature_sensor_entity.temperature.value}) # close the mqtt listening thread @@ -243,20 +247,22 @@ # disconnect the mqtt device mqttc.disconnect() - # plot results + # plot the results fig, ax = plt.subplots() - t_simulation = [item["simtime"] for item in history_weather_station] + t_simulation = [item["sim_time"]/3600 for item in history_weather_station] temperature = [item["temperature"] for item in history_weather_station] ax.plot(t_simulation, temperature) - ax.set_xlabel('time in s') + ax.title.set_text("Weather Station") + ax.set_xlabel('time in h') ax.set_ylabel('ambient temperature in °C') fig2, ax2 = plt.subplots() - t_simulation = [item["simtime"] for item in history_zone_temperature_sensor] + t_simulation = [item["sim_time"]/3600 for item in history_zone_temperature_sensor] temperature = [item["temperature"] for item in history_zone_temperature_sensor] ax2.plot(t_simulation, temperature) - ax2.set_xlabel('time in s') + ax2.title.set_text("Zone Temperature Sensor") + ax2.set_xlabel('time in h') ax2.set_ylabel('zone temperature in °C') plt.show() @@ -266,15 +272,15 @@ f"Wrong file extension! {WRITE_DEVICES_FILEPATH.suffix}" WRITE_DEVICES_FILEPATH.touch(exist_ok=True) with WRITE_DEVICES_FILEPATH.open('w', encoding='utf-8') as f: - devices = [item.dict() for item in iotac.get_device_list()] + devices = [item.model_dump() for item in iotac.get_device_list()] json.dump(devices, f, ensure_ascii=False, indent=2) assert WRITE_GROUPS_FILEPATH.suffix == '.json', \ f"Wrong file extension! {WRITE_GROUPS_FILEPATH.suffix}" WRITE_GROUPS_FILEPATH.touch(exist_ok=True) with WRITE_GROUPS_FILEPATH.open('w', encoding='utf-8') as f: - groups = [item.dict() for item in iotac.get_group_list()] + groups = [item.model_dump() for item in iotac.get_group_list()] json.dump(groups, f, ensure_ascii=False, indent=2) - clear_context_broker(url=CB_URL, fiware_header=fiware_header) clear_iot_agent(url=IOTA_URL, fiware_header=fiware_header) + clear_context_broker(url=CB_URL, fiware_header=fiware_header) diff --git a/tutorials/ngsi_v2/e5_iot_thermal_zone_control/e5_iot_thermal_zone_control.py b/tutorials/ngsi_v2/e5_iot_thermal_zone_control/e5_iot_thermal_zone_control.py index 0b9be634..ff250032 100644 --- a/tutorials/ngsi_v2/e5_iot_thermal_zone_control/e5_iot_thermal_zone_control.py +++ b/tutorials/ngsi_v2/e5_iot_thermal_zone_control/e5_iot_thermal_zone_control.py @@ -3,7 +3,7 @@ # Create a virtual IoT device that simulates a heater for your # thermal zone. The heater can be turned on and off via a simple hysteresis -# controller. The devices from e4_iot_thermal_zone_sensors.py will loaded +# controller. The devices from e4_iot_thermal_zone_sensors.py will be loaded # from the stored *.json-files. # The input sections are marked with 'ToDo' @@ -13,18 +13,18 @@ # 2. Retrieve the service group and device configurations of already existing # devices from the IoT-Agent # 3. Create a third device configuration for a heater holding a command -# for turning it `on` and `off`device and post it to the server +# for turning it `on` and `off` and post it to the server # 4. Create an MQTT client using the filip.client.mqtt package and register # your service group and your devices # 4. Define a callback function that will be executed when the client -# receives a command. Decode the message and set the update the state in +# receives a command. Decode the message and set the update state in # simulation model. Afterwards, acknowledge the command using the api of the # IoTAMQTTClient. # 5. Add the callback for your heater device to the IoTAMQTTClient # 6. Create an MQTT subscription for asynchronous communication that # gets triggered when the temperature attribute changes. # 7. Write a second callback that represents your controller. It should get -# triggered when the MQTTClient receive a notification message due to your +# triggered when the MQTTClient receives a notification message due to your # subscription. Add the callback to your MQTTClient using the original # paho-api (`message_callback_add`) # 8. Run the simulation and plot @@ -38,13 +38,13 @@ from urllib.parse import urlparse from uuid import uuid4 import paho.mqtt.client as mqtt -from pydantic import parse_file_as +from pydantic import TypeAdapter import matplotlib.pyplot as plt # import from filip from filip.clients.ngsi_v2 import ContextBrokerClient, IoTAClient from filip.clients.mqtt import IoTAMQTTClient -from filip.models.base import FiwareHeader +from filip.models.base import DataType, FiwareHeader from filip.models.ngsi_v2.context import NamedCommand from filip.models.ngsi_v2.subscriptions import Subscription, Message from filip.models.ngsi_v2.iot import \ @@ -58,42 +58,53 @@ from tutorials.ngsi_v2.simulation_model import SimulationModel # ## Parameters -# ToDo: Enter your context broker host and port, e.g http://localhost:1026 +# ToDo: Enter your context broker host and port, e.g http://localhost:1026. CB_URL = "http://localhost:1026" -# ToDo: Enter your IoT-Agent host and port, e.g http://localhost:4041 +# ToDo: Enter your IoT-Agent host and port, e.g http://localhost:4041. IOTA_URL = "http://localhost:4041" -# ToDo: Enter your mqtt broker url, e.g mqtt://test.mosquitto.org:1883 +# ToDo: Enter your mqtt broker url, e.g mqtt://test.mosquitto.org:1883. MQTT_BROKER_URL_EXPOSED = "mqtt://localhost:1883" -# ToDo: Enter your mqtt broker url, e.g mqtt://mosquitto:1883 +# ToDo: Enter your mqtt broker url, e.g mqtt://mosquitto:1883. MQTT_BROKER_URL_INTERNAL = "mqtt://mosquitto:1883" -# ToDo: If required enter your username and password +# ToDo: If required, enter your username and password. MQTT_USER = "" -MQTT_PW = "" +MQTT_PW = "" + +# ToDo: Change the name of your service to something unique. If you run +# on a shared instance this is very important in order to avoid user +# collisions. You will use this service through the whole tutorial. +# If you forget to change it, an error will be raised! # FIWARE-Service SERVICE = 'filip_tutorial' -# FIWARE-Servicepath -# ToDo: Change the name of your service-path to something unique. If you run -# on a shared instance this very important in order to avoid user -# collisions. You will use this service path through the whole tutorial. -# If you forget to change it an error will be raised! -SERVICE_PATH = '/' -APIKEY = SERVICE_PATH.strip('/') +# FIWARE-Service path +SERVICE_PATH = '/' + +# ToDo: Change the APIKEY to something unique. This represents the "token" +# for IoT devices to connect (send/receive data) with the platform. In the +# context of MQTT, APIKEY is linked with the topic used for communication. +APIKEY = 'your_apikey' UNIQUE_ID = str(uuid4()) TOPIC_CONTROLLER = f"fiware_workshop/{UNIQUE_ID}/controller" print(TOPIC_CONTROLLER) -# Path to json-files to store entity data for follow up exercises +# path to json-files to store entity data for follow-up exercises WRITE_GROUPS_FILEPATH = \ Path("../e5_iot_thermal_zone_control_groups.json") WRITE_DEVICES_FILEPATH = \ Path("../e5_iot_thermal_zone_control_devices.json") WRITE_SUBSCRIPTIONS_FILEPATH = \ Path("../e5_iot_thermal_zone_control_subscriptions.json") -# Path to read json-files from previous exercises +# path to read json-files from previous exercises READ_GROUPS_FILEPATH = \ Path("../e4_iot_thermal_zone_sensors_groups.json") READ_DEVICES_FILEPATH = \ Path("../e4_iot_thermal_zone_sensors_devices.json") +# opening the files +with open(READ_GROUPS_FILEPATH, 'r') as groups_file, \ + open(READ_DEVICES_FILEPATH, 'r') as devices_file: + json_groups = json.load(groups_file) + json_devices = json.load(devices_file) + # set parameters for the temperature simulation TEMPERATURE_MAX = 10 # maximal ambient temperature TEMPERATURE_MIN = -5 # minimal ambient temperature @@ -101,7 +112,7 @@ T_SIM_START = 0 # simulation start time in seconds T_SIM_END = 24 * 60 * 60 # simulation end time in seconds -COM_STEP = 60 * 60 * 0.25 # 15 min communication step in seconds +COM_STEP = 60 * 60 * 0.25 # 15 min communication step in seconds # ## Main script if __name__ == '__main__': @@ -124,34 +135,34 @@ history_zone_temperature_sensor = [] history_heater = [] - # Create clients and restore devices and groups from file - groups = parse_file_as(List[ServiceGroup], READ_GROUPS_FILEPATH) - devices = parse_file_as(List[Device], READ_DEVICES_FILEPATH) + # create clients and also restore devices and groups from file + groups = TypeAdapter(List[ServiceGroup]).validate_python(json_groups) + devices = TypeAdapter(List[Device]).validate_python(json_devices) cbc = ContextBrokerClient(url=CB_URL, fiware_header=fiware_header) iotac = IoTAClient(url=IOTA_URL, fiware_header=fiware_header) iotac.post_groups(service_groups=groups) iotac.post_devices(devices=devices) - # ToDo: Get the device configurations from the server + # ToDo: Get the device configurations from the server. weather_station = iotac.get_device(device_id="device:001") zone_temperature_sensor = ... - # ToDo: Get the service group configurations from the server + # ToDo: Get the service group configurations from the server. group = iotac.get_group(resource="/iot/json", apikey=...) - # ToDo: Create and additional device holding a command attribute and - # post it to the IoT-Agent. It should be mapped to the `type` heater - # create the simtime attribute and add during device creation - t_sim = DeviceAttribute(name='simtime', + # ToDo: Create an additional device holding a command attribute and + # post it to the IoT-Agent. It should be mapped to the `type` heater. + # create the sim_time attribute and add it during device creation + t_sim = DeviceAttribute(name='sim_time', object_id='t_sim', type="Number") - # ToDo: create the command attribute of name `heater_on` (currently it is - # not possible to add metadata here) + # ToDo: Create the command attribute of name `heater_on` (currently it is + # not possible to add metadata here). cmd = DeviceCommand(name=..., type=...) - # ToDo: create the device configuration and send it to the server name it + # ToDo: Create the device configuration and send it to the server. heater = Device(...) @@ -163,11 +174,11 @@ iotac.post_device(device=heater) - # ToDo: Check the entity that corresponds to your device + # ToDo: Check the entity that corresponds to your device. heater_entity = cbc.get_entity(entity_id=heater.entity_name, entity_type=heater.entity_type) print(f"Your device entity before running the simulation: \n " - f"{heater_entity.json(indent=2)}") + f"{heater_entity.model_dump_json(indent=2)}") # create a MQTTv5 client with paho-mqtt and the known groups and devices. mqttc = IoTAMQTTClient(protocol=mqtt.MQTTv5, @@ -179,27 +190,27 @@ mqttc.username_pw_set(username=MQTT_USER, password=MQTT_PW) # ToDo: Implement a callback function that gets triggered when the - # command is sent to the device. The incoming command schould update the - # heater attribute of the simulation model + # command is sent to the device. The incoming command should update the + # heater attribute of the simulation model. def on_command(client, obj, msg): """ Callback for incoming commands """ - # Decode the message payload using the libraries builtin encoders + # decode the message payload using the libraries builtin encoders apikey, device_id, payload = \ client.get_encoder(PayloadProtocol.IOTA_JSON).decode_message( msg=msg) # map the command value to the simulation sim_model.heater_on = payload[cmd.name] - # ToDo: acknowledge the command. Here command are usually single + # ToDo: Acknowledge the command. In this case commands are usually single # messages. The first key is equal to the commands name. client.publish(device_id=device_id, command_name=..., payload=...) # ToDo: Add the command callback to your MQTTClient. This will get - # triggered for the specified device_id + # triggered for the specified device_id. mqttc.add_command_callback(device_id=..., callback=...) @@ -226,26 +237,23 @@ def on_command(client, obj, msg): }, "throttling": 0 } - # Generate Subscription object for validation and post it + # generate Subscription object for validation and post it subscription = Subscription(**subscription) subscription_id = cbc.post_subscription(subscription=subscription) # ToDo: You need to implement a controller that controls the # heater state with respect to the zone temperature. This will be - # implemented with asynchronous communication using MQTT-Subscriptions + # implemented with asynchronous communication using MQTT-Subscriptions. def on_measurement(client, obj, msg): """ Callback for measurement notifications """ - message = Message.parse_raw(msg.payload) + message = Message.model_validate_json(msg.payload) updated_zone_temperature_sensor = message.data[0] - # ToDo: retrieve the value of temperature attribute + # ToDo: Retrieve the value of temperature attribute. temperature = ... - # ToDo: device if you want update your command - # Note that this could also be substitute by a conditional - # subscription update = True if temperature <= 19: state = 1 @@ -253,7 +261,8 @@ def on_measurement(client, obj, msg): state = 0 else: update = False - # ToDo: send the command to the heater entity + + # ToDo: Send the command to the heater entity. if update: command = NamedCommand(name=cmd.name, value=state) cbc.post_command(...) @@ -280,9 +289,9 @@ def on_measurement(client, obj, msg): # create a non-blocking thread for mqtt communication mqttc.loop_start() - # Create a loop that publishes every second a message to the broker - # that holds the simulation time "simtime" and the corresponding - # temperature "temperature" the loop should. You may use the `object_id` + # ToDo: Create a loop that publishes a message every 100 milliseconds + # to the broker that holds the simulation time "sim_time" and the + # corresponding temperature "temperature". You may use the `object_id` # or the attribute name as key in your payload. for t_sim in range(sim_model.t_start, sim_model.t_end + int(COM_STEP), @@ -290,48 +299,48 @@ def on_measurement(client, obj, msg): # publish the simulated ambient temperature mqttc.publish(device_id=weather_station.device_id, payload={"temperature": sim_model.t_amb, - "simtime": sim_model.t_sim}) + "sim_time": sim_model.t_sim}) # publish the simulated zone temperature mqttc.publish(device_id=zone_temperature_sensor.device_id, payload={"temperature": sim_model.t_zone, - "simtime": sim_model.t_sim}) + "sim_time": sim_model.t_sim}) - # publish the 'simtime' for the heater device + # publish the 'sim_time' for the heater device mqttc.publish(device_id=heater.device_id, - payload={"simtime": sim_model.t_sim}) + payload={"sim_time": sim_model.t_sim}) - time.sleep(0.5) + time.sleep(0.1) # simulation step for next loop sim_model.do_step(int(t_sim + COM_STEP)) - # wait for one second before publishing the next values - time.sleep(0.5) + # wait for 0.1 second before publishing the next values + time.sleep(0.1) - # Get corresponding entities and write values to history + # get corresponding entities and write values to history weather_station_entity = cbc.get_entity( entity_id=weather_station.entity_name, entity_type=weather_station.entity_type ) # append the data to the local history history_weather_station.append( - {"simtime": weather_station_entity.simtime.value, + {"sim_time": weather_station_entity.sim_time.value, "temperature": weather_station_entity.temperature.value}) - # Get ZoneTemperatureSensor and write values to history + # get zone temperature sensor and write values to history zone_temperature_sensor_entity = cbc.get_entity( entity_id=zone_temperature_sensor.entity_name, entity_type=zone_temperature_sensor.entity_type ) history_zone_temperature_sensor.append( - {"simtime": zone_temperature_sensor_entity.simtime.value, + {"sim_time": zone_temperature_sensor_entity.sim_time.value, "temperature": zone_temperature_sensor_entity.temperature.value}) - # Get ZoneTemperatureSensor and write values to history + # get zone temperature sensor and write values to history heater_entity = cbc.get_entity( entity_id=heater.entity_name, entity_type=heater.entity_type) history_heater.append( - {"simtime": heater_entity.simtime.value, + {"sim_time": heater_entity.sim_time.value, "on_off": heater_entity.heater_on_info.value}) # close the mqtt listening thread @@ -340,31 +349,35 @@ def on_measurement(client, obj, msg): mqttc.disconnect() print(cbc.get_entity(entity_id=heater.entity_name, - entity_type=heater.entity_type).json(indent=2)) + entity_type=heater.entity_type).model_dump_json(indent=2)) # plot results fig, ax = plt.subplots() - t_simulation = [item["simtime"] for item in history_weather_station] + t_simulation = [item["sim_time"]/60 for item in history_weather_station] temperature = [item["temperature"] for item in history_weather_station] ax.plot(t_simulation, temperature) - ax.set_xlabel('time in s') + ax.title.set_text("Weather Station") + ax.set_xlabel('time in min') ax.set_ylabel('ambient temperature in °C') + plt.show() fig2, ax2 = plt.subplots() - t_simulation = [item["simtime"] for item in history_zone_temperature_sensor] + t_simulation = [item["sim_time"]/60 for item in history_zone_temperature_sensor] temperature = [item["temperature"] for item in history_zone_temperature_sensor] ax2.plot(t_simulation, temperature) - ax2.set_xlabel('time in s') + ax2.title.set_text("Zone Temperature Sensor") + ax2.set_xlabel('time in min') ax2.set_ylabel('zone temperature in °C') + plt.show() fig3, ax3 = plt.subplots() - t_simulation = [item["simtime"] for item in history_heater] + t_simulation = [item["sim_time"]/60 for item in history_heater] on_off = [item["on_off"] for item in history_heater] ax3.plot(t_simulation, on_off) - ax3.set_xlabel('time in s') + ax3.title.set_text("Heater") + ax3.set_xlabel('time in min') ax3.set_ylabel('on/off') - plt.show() # write devices and groups to file and clear server state @@ -372,21 +385,21 @@ def on_measurement(client, obj, msg): f"Wrong file extension! {WRITE_DEVICES_FILEPATH.suffix}" WRITE_DEVICES_FILEPATH.touch(exist_ok=True) with WRITE_DEVICES_FILEPATH.open('w', encoding='utf-8') as f: - devices = [item.dict() for item in iotac.get_device_list()] + devices = [item.model_dump() for item in iotac.get_device_list()] json.dump(devices, f, ensure_ascii=False, indent=2) assert WRITE_GROUPS_FILEPATH.suffix == '.json', \ f"Wrong file extension! {WRITE_GROUPS_FILEPATH.suffix}" WRITE_GROUPS_FILEPATH.touch(exist_ok=True) with WRITE_GROUPS_FILEPATH.open('w', encoding='utf-8') as f: - groups = [item.dict() for item in iotac.get_group_list()] + groups = [item.model_dump() for item in iotac.get_group_list()] json.dump(groups, f, ensure_ascii=False, indent=2) assert WRITE_SUBSCRIPTIONS_FILEPATH.suffix == '.json', \ f"Wrong file extension! {WRITE_SUBSCRIPTIONS_FILEPATH.suffix}" WRITE_SUBSCRIPTIONS_FILEPATH.touch(exist_ok=True) with WRITE_SUBSCRIPTIONS_FILEPATH.open('w', encoding='utf-8') as f: - subs = [item.dict() for item in cbc.get_subscription_list()] + subs = [item.model_dump() for item in cbc.get_subscription_list()] json.dump(subs, f, ensure_ascii=False, indent=2) clear_iot_agent(url=IOTA_URL, fiware_header=fiware_header) diff --git a/tutorials/ngsi_v2/e5_iot_thermal_zone_control/e5_iot_thermal_zone_control_solution.py b/tutorials/ngsi_v2/e5_iot_thermal_zone_control/e5_iot_thermal_zone_control_solution.py index 66219210..d05bccdf 100644 --- a/tutorials/ngsi_v2/e5_iot_thermal_zone_control/e5_iot_thermal_zone_control_solution.py +++ b/tutorials/ngsi_v2/e5_iot_thermal_zone_control/e5_iot_thermal_zone_control_solution.py @@ -3,7 +3,7 @@ # Create a virtual IoT device that simulates a heater for your # thermal zone. The heater can be turned on and off via a simple hysteresis -# controller. The devices from e4_iot_thermal_zone_sensors.py will loaded +# controller. The devices from e4_iot_thermal_zone_sensors.py will be loaded # from the stored *.json-files. # The input sections are marked with 'ToDo' @@ -13,18 +13,18 @@ # 2. Retrieve the service group and device configurations of already existing # devices from the IoT-Agent # 3. Create a third device configuration for a heater holding a command -# for turning it `on` and `off`device and post it to the server +# for turning it `on` and `off` and post it to the server # 4. Create an MQTT client using the filip.client.mqtt package and register # your service group and your devices # 4. Define a callback function that will be executed when the client -# receives a command. Decode the message and set the update the state in +# receives a command. Decode the message and set the update state in # simulation model. Afterwards, acknowledge the command using the api of the # IoTAMQTTClient. # 5. Add the callback for your heater device to the IoTAMQTTClient # 6. Create an MQTT subscription for asynchronous communication that # gets triggered when the temperature attribute changes. # 7. Write a second callback that represents your controller. It should get -# triggered when the MQTTClient receive a notification message due to your +# triggered when the MQTTClient receives a notification message due to your # subscription. Add the callback to your MQTTClient using the original # paho-api (`message_callback_add`) # 8. Run the simulation and plot @@ -38,7 +38,7 @@ from urllib.parse import urlparse from uuid import uuid4 import paho.mqtt.client as mqtt -from pydantic import parse_file_as +from pydantic import TypeAdapter import matplotlib.pyplot as plt # import from filip @@ -58,42 +58,53 @@ from tutorials.ngsi_v2.simulation_model import SimulationModel # ## Parameters -# ToDo: Enter your context broker host and port, e.g http://localhost:1026 +# ToDo: Enter your context broker host and port, e.g http://localhost:1026. CB_URL = "http://localhost:1026" -# ToDo: Enter your IoT-Agent host and port, e.g http://localhost:4041 +# ToDo: Enter your IoT-Agent host and port, e.g http://localhost:4041. IOTA_URL = "http://localhost:4041" -# ToDo: Enter your mqtt broker url, e.g mqtt://test.mosquitto.org:1883 +# ToDo: Enter your mqtt broker url, e.g mqtt://test.mosquitto.org:1883. MQTT_BROKER_URL_EXPOSED = "mqtt://localhost:1883" -# ToDo: Enter your mqtt broker url, e.g mqtt://mosquitto:1883 +# ToDo: Enter your mqtt broker url, e.g mqtt://mosquitto:1883. MQTT_BROKER_URL_INTERNAL = "mqtt://mosquitto:1883" -# ToDo: If required enter your username and password +# ToDo: If required, enter your username and password. MQTT_USER = "" -MQTT_PW = "" +MQTT_PW = "" + +# ToDo: Change the name of your service to something unique. If you run +# on a shared instance this is very important in order to avoid user +# collisions. You will use this service through the whole tutorial. +# If you forget to change it, an error will be raised! # FIWARE-Service SERVICE = 'filip_tutorial' -# FIWARE-Servicepath -# ToDo: Change the name of your service-path to something unique. If you run -# on a shared instance this very important in order to avoid user -# collisions. You will use this service path through the whole tutorial. -# If you forget to change it an error will be raised! -SERVICE_PATH = '/your_path' -APIKEY = SERVICE_PATH.strip('/') +# FIWARE-Service path +SERVICE_PATH = '/' + +# ToDo: Change the APIKEY to something unique. This represents the "token" +# for IoT devices to connect (send/receive data) with the platform. In the +# context of MQTT, APIKEY is linked with the topic used for communication. +APIKEY = 'your_apikey' UNIQUE_ID = str(uuid4()) TOPIC_CONTROLLER = f"fiware_workshop/{UNIQUE_ID}/controller" print(TOPIC_CONTROLLER) -# Path to json-files to store entity data for follow up exercises +# path to json-files to store entity data for follow-up exercises WRITE_GROUPS_FILEPATH = \ Path("../e5_iot_thermal_zone_control_solution_groups.json") WRITE_DEVICES_FILEPATH = \ Path("../e5_iot_thermal_zone_control_solution_devices.json") WRITE_SUBSCRIPTIONS_FILEPATH = \ Path("../e5_iot_thermal_zone_control_solution_subscriptions.json") -# Path to read json-files from previous exercises +# path to read json-files from previous exercises READ_GROUPS_FILEPATH = \ Path("../e4_iot_thermal_zone_sensors_solution_groups.json") READ_DEVICES_FILEPATH = \ Path("../e4_iot_thermal_zone_sensors_solution_devices.json") +# opening the files +with open(READ_GROUPS_FILEPATH, 'r') as groups_file, \ + open(READ_DEVICES_FILEPATH, 'r') as devices_file: + json_groups = json.load(groups_file) + json_devices = json.load(devices_file) + # set parameters for the temperature simulation TEMPERATURE_MAX = 10 # maximal ambient temperature TEMPERATURE_MIN = -5 # minimal ambient temperature @@ -101,7 +112,7 @@ T_SIM_START = 0 # simulation start time in seconds T_SIM_END = 24 * 60 * 60 # simulation end time in seconds -COM_STEP = 60 * 60 * 0.25 # 15 min communication step in seconds +COM_STEP = 60 * 60 * 0.25 # 15 min communication step in seconds # ## Main script if __name__ == '__main__': @@ -123,35 +134,35 @@ history_weather_station = [] history_zone_temperature_sensor = [] history_heater = [] - - # Create clients and restore devices and groups from file - groups = parse_file_as(List[ServiceGroup], READ_GROUPS_FILEPATH) - devices = parse_file_as(List[Device], READ_DEVICES_FILEPATH) + + # create clients and also restore devices and groups from file + groups = TypeAdapter(List[ServiceGroup]).validate_python(json_groups) + devices = TypeAdapter(List[Device]).validate_python(json_devices) cbc = ContextBrokerClient(url=CB_URL, fiware_header=fiware_header) iotac = IoTAClient(url=IOTA_URL, fiware_header=fiware_header) iotac.post_groups(service_groups=groups) iotac.post_devices(devices=devices) - # ToDo: Get the device configurations from the server + # ToDo: Get the device configurations from the server. weather_station = iotac.get_device(device_id="device:001") zone_temperature_sensor = iotac.get_device(device_id="device:002") - # ToDo: Get the service group configurations from the server + # ToDo: Get the service group configurations from the server. group = iotac.get_group(resource="/iot/json", apikey=APIKEY) - # ToDo: Create and additional device holding a command attribute and - # post it to the IoT-Agent. It should be mapped to the `type` heater - # create the simtime attribute and add during device creation - t_sim = DeviceAttribute(name='simtime', + # ToDo: Create an additional device holding a command attribute and + # post it to the IoT-Agent. It should be mapped to the `type` heater. + # create the sim_time attribute and add it during device creation + t_sim = DeviceAttribute(name='sim_time', object_id='t_sim', type="Number") - # ToDo: create the command attribute of name `heater_on` (currently it is - # not possible to add metadata here) + # ToDo: Create the command attribute of name `heater_on` (currently it is + # not possible to add metadata here). cmd = DeviceCommand(name="heater_on", type=DataType.BOOLEAN) - # ToDo: create the device configuration and send it to the server + # ToDo: Create the device configuration and send it to the server. heater = Device(device_id="device:003", entity_name="urn:ngsi-ld:Heater:001", entity_type="Heater", @@ -163,11 +174,11 @@ iotac.post_device(device=heater) - # ToDo: Check the entity that corresponds to your device + # ToDo: Check the entity that corresponds to your device. heater_entity = cbc.get_entity(entity_id=heater.entity_name, entity_type=heater.entity_type) print(f"Your device entity before running the simulation: \n " - f"{heater_entity.json(indent=2)}") + f"{heater_entity.model_dump_json(indent=2)}") # create a MQTTv5 client with paho-mqtt and the known groups and devices. mqttc = IoTAMQTTClient(protocol=mqtt.MQTTv5, @@ -179,27 +190,27 @@ mqttc.username_pw_set(username=MQTT_USER, password=MQTT_PW) # ToDo: Implement a callback function that gets triggered when the - # command is sent to the device. The incoming command schould update the - # heater attribute of the simulation model + # command is sent to the device. The incoming command should update the + # heater attribute of the simulation model. def on_command(client, obj, msg): """ Callback for incoming commands """ - # Decode the message payload using the libraries builtin encoders + # decode the message payload using the libraries builtin encoders apikey, device_id, payload = \ client.get_encoder(PayloadProtocol.IOTA_JSON).decode_message( msg=msg) # map the command value to the simulation sim_model.heater_on = payload[cmd.name] - # ToDo: acknowledge the command. Here command are usually single + # ToDo: Acknowledge the command. In this case commands are usually single # messages. The first key is equal to the commands name. client.publish(device_id=device_id, command_name=next(iter(payload)), payload=payload) # ToDo: Add the command callback to your MQTTClient. This will get - # triggered for the specified device_id + # triggered for the specified device_id. mqttc.add_command_callback(device_id=heater.device_id, callback=on_command) @@ -226,26 +237,23 @@ def on_command(client, obj, msg): }, "throttling": 0 } - # Generate Subscription object for validation and post it + # generate Subscription object for validation and post it subscription = Subscription(**subscription) subscription_id = cbc.post_subscription(subscription=subscription) # ToDo: You need to implement a controller that controls the # heater state with respect to the zone temperature. This will be - # implemented with asynchronous communication using MQTT-Subscriptions + # implemented with asynchronous communication using MQTT-Subscriptions. def on_measurement(client, obj, msg): """ Callback for measurement notifications """ - message = Message.parse_raw(msg.payload) + message = Message.model_validate_json(msg.payload) updated_zone_temperature_sensor = message.data[0] - # ToDo: retrieve the value of temperature attribute + # ToDo: Retrieve the value of temperature attribute. temperature = updated_zone_temperature_sensor.temperature.value - # ToDo: device if you want update your command - # Note that this could also be substitute by a conditional - # subscription update = True if temperature <= 19: state = 1 @@ -253,7 +261,8 @@ def on_measurement(client, obj, msg): state = 0 else: update = False - # ToDo: send the command to the heater entity + + # ToDo: Send the command to the heater entity. if update: command = NamedCommand(name=cmd.name, value=state) cbc.post_command(entity_id=heater.entity_name, @@ -280,9 +289,9 @@ def on_measurement(client, obj, msg): # create a non-blocking thread for mqtt communication mqttc.loop_start() - # Create a loop that publishes every second a message to the broker - # that holds the simulation time "simtime" and the corresponding - # temperature "temperature" the loop should. You may use the `object_id` + # ToDo: Create a loop that publishes a message every 100 milliseconds + # to the broker that holds the simulation time "sim_time" and the + # corresponding temperature "temperature". You may use the `object_id` # or the attribute name as key in your payload. for t_sim in range(sim_model.t_start, sim_model.t_end + int(COM_STEP), @@ -290,48 +299,48 @@ def on_measurement(client, obj, msg): # publish the simulated ambient temperature mqttc.publish(device_id=weather_station.device_id, payload={"temperature": sim_model.t_amb, - "simtime": sim_model.t_sim}) + "sim_time": sim_model.t_sim}) # publish the simulated zone temperature mqttc.publish(device_id=zone_temperature_sensor.device_id, payload={"temperature": sim_model.t_zone, - "simtime": sim_model.t_sim}) + "sim_time": sim_model.t_sim}) - # publish the 'simtime' for the heater device + # publish the 'sim_time' for the heater device mqttc.publish(device_id=heater.device_id, - payload={"simtime": sim_model.t_sim}) + payload={"sim_time": sim_model.t_sim}) - time.sleep(0.5) + time.sleep(0.1) # simulation step for next loop sim_model.do_step(int(t_sim + COM_STEP)) - # wait for one second before publishing the next values - time.sleep(0.5) + # wait for 0.1 second before publishing the next values + time.sleep(0.1) - # Get corresponding entities and write values to history + # get corresponding entities and write values to history weather_station_entity = cbc.get_entity( entity_id=weather_station.entity_name, entity_type=weather_station.entity_type ) # append the data to the local history history_weather_station.append( - {"simtime": weather_station_entity.simtime.value, + {"sim_time": weather_station_entity.sim_time.value, "temperature": weather_station_entity.temperature.value}) - # Get ZoneTemperatureSensor and write values to history + # get zone temperature sensor and write values to history zone_temperature_sensor_entity = cbc.get_entity( entity_id=zone_temperature_sensor.entity_name, entity_type=zone_temperature_sensor.entity_type ) history_zone_temperature_sensor.append( - {"simtime": zone_temperature_sensor_entity.simtime.value, + {"sim_time": zone_temperature_sensor_entity.sim_time.value, "temperature": zone_temperature_sensor_entity.temperature.value}) - # Get ZoneTemperatureSensor and write values to history + # get zone temperature sensor and write values to history heater_entity = cbc.get_entity( entity_id=heater.entity_name, entity_type=heater.entity_type) history_heater.append( - {"simtime": heater_entity.simtime.value, + {"sim_time": heater_entity.sim_time.value, "on_off": heater_entity.heater_on_info.value}) # close the mqtt listening thread @@ -340,31 +349,35 @@ def on_measurement(client, obj, msg): mqttc.disconnect() print(cbc.get_entity(entity_id=heater.entity_name, - entity_type=heater.entity_type).json(indent=2)) + entity_type=heater.entity_type).model_dump_json(indent=2)) # plot results fig, ax = plt.subplots() - t_simulation = [item["simtime"] for item in history_weather_station] + t_simulation = [item["sim_time"]/60 for item in history_weather_station] temperature = [item["temperature"] for item in history_weather_station] ax.plot(t_simulation, temperature) - ax.set_xlabel('time in s') + ax.title.set_text("Weather Station") + ax.set_xlabel('time in min') ax.set_ylabel('ambient temperature in °C') + plt.show() fig2, ax2 = plt.subplots() - t_simulation = [item["simtime"] for item in history_zone_temperature_sensor] + t_simulation = [item["sim_time"]/60 for item in history_zone_temperature_sensor] temperature = [item["temperature"] for item in history_zone_temperature_sensor] ax2.plot(t_simulation, temperature) - ax2.set_xlabel('time in s') + ax2.title.set_text("Zone Temperature Sensor") + ax2.set_xlabel('time in min') ax2.set_ylabel('zone temperature in °C') + plt.show() fig3, ax3 = plt.subplots() - t_simulation = [item["simtime"] for item in history_heater] + t_simulation = [item["sim_time"]/60 for item in history_heater] on_off = [item["on_off"] for item in history_heater] ax3.plot(t_simulation, on_off) - ax3.set_xlabel('time in s') + ax3.title.set_text("Heater") + ax3.set_xlabel('time in min') ax3.set_ylabel('on/off') - plt.show() # write devices and groups to file and clear server state @@ -372,21 +385,21 @@ def on_measurement(client, obj, msg): f"Wrong file extension! {WRITE_DEVICES_FILEPATH.suffix}" WRITE_DEVICES_FILEPATH.touch(exist_ok=True) with WRITE_DEVICES_FILEPATH.open('w', encoding='utf-8') as f: - devices = [item.dict() for item in iotac.get_device_list()] + devices = [item.model_dump() for item in iotac.get_device_list()] json.dump(devices, f, ensure_ascii=False, indent=2) assert WRITE_GROUPS_FILEPATH.suffix == '.json', \ f"Wrong file extension! {WRITE_GROUPS_FILEPATH.suffix}" WRITE_GROUPS_FILEPATH.touch(exist_ok=True) with WRITE_GROUPS_FILEPATH.open('w', encoding='utf-8') as f: - groups = [item.dict() for item in iotac.get_group_list()] + groups = [item.model_dump() for item in iotac.get_group_list()] json.dump(groups, f, ensure_ascii=False, indent=2) assert WRITE_SUBSCRIPTIONS_FILEPATH.suffix == '.json', \ f"Wrong file extension! {WRITE_SUBSCRIPTIONS_FILEPATH.suffix}" WRITE_SUBSCRIPTIONS_FILEPATH.touch(exist_ok=True) with WRITE_SUBSCRIPTIONS_FILEPATH.open('w', encoding='utf-8') as f: - subs = [item.dict() for item in cbc.get_subscription_list()] + subs = [item.model_dump() for item in cbc.get_subscription_list()] json.dump(subs, f, ensure_ascii=False, indent=2) clear_iot_agent(url=IOTA_URL, fiware_header=fiware_header) diff --git a/tutorials/ngsi_v2/e6_timeseries_data/e6_timeseries_data.py b/tutorials/ngsi_v2/e6_timeseries_data/e6_timeseries_data.py index e0513cd2..3e7e9174 100644 --- a/tutorials/ngsi_v2/e6_timeseries_data/e6_timeseries_data.py +++ b/tutorials/ngsi_v2/e6_timeseries_data/e6_timeseries_data.py @@ -7,13 +7,14 @@ # #### Steps to complete: # 1. Set up the missing parameters in the parameter section -# 2. Create a quantumleap client that create subscription that get triggered +# 2. Create a quantumleap client that creates subscription that gets triggered # by the updates on your context entities # 3. Run the simulation # 4. Retrieve the data via QuantumLeap and visualize it """ # ## Import packages +import json from pathlib import Path import time from typing import List @@ -21,9 +22,8 @@ from uuid import uuid4 import pandas as pd import paho.mqtt.client as mqtt -from pydantic import parse_file_as +from pydantic import TypeAdapter import matplotlib.pyplot as plt - # import from filip from filip.clients.ngsi_v2 import \ ContextBrokerClient, \ @@ -32,7 +32,8 @@ from filip.clients.mqtt import IoTAMQTTClient from filip.models.base import FiwareHeader from filip.models.ngsi_v2.context import NamedCommand -from filip.models.ngsi_v2.subscriptions import Subscription, Message +from filip.models.ngsi_v2.subscriptions import Subscription, Message, Subject, \ + Notification from filip.models.ngsi_v2.iot import \ Device, \ PayloadProtocol, \ @@ -45,40 +46,52 @@ from tutorials.ngsi_v2.simulation_model import SimulationModel # ## Parameters -# ToDo: Enter your context broker host and port, e.g http://localhost:1026 +# ToDo: Enter your context broker host and port, e.g. http://localhost:1026. CB_URL = "http://localhost:1026" -# ToDo: Enter your IoT-Agent host and port, e.g http://localhost:4041 +# ToDo: Enter your IoT-Agent host and port, e.g. http://localhost:4041. IOTA_URL = "http://localhost:4041" -# ToDo: Enter your IoT-Agent host and port, e.g http://localhost:4041 +# ToDo: Enter your IoT-Agent host and port, e.g. http://localhost:4041. QL_URL = "http://localhost:8668" -# ToDo: Enter your mqtt broker url, e.g mqtt://test.mosquitto.org:1883 +# ToDo: Enter your mqtt broker url, e.g. mqtt://test.mosquitto.org:1883. MQTT_BROKER_URL_EXPOSED = "mqtt://localhost:1883" -# ToDo: Enter your mqtt broker url, e.g mqtt://mosquitto:1883 +# ToDo: Enter your mqtt broker url, e.g. mqtt://mosquitto:1883. MQTT_BROKER_URL_INTERNAL = "mqtt://mosquitto:1883" -# ToDo: If required enter your username and password +# ToDo: If required, enter your username and password. MQTT_USER = "" -MQTT_PW = "" +MQTT_PW = "" +# ToDo: Change the name of your service to something unique. If you run +# on a shared instance this is very important in order to avoid user +# collisions. You will use this service through the whole tutorial. +# If you forget to change it, an error will be raised! # FIWARE-Service SERVICE = 'filip_tutorial' -# FIWARE-Servicepath -# ToDo: Change the name of your service-path to something unique. If you run -# on a shared instance this very important in order to avoid user -# collisions. You will use this service path through the whole tutorial. -# If you forget to change it an error will be raised! -SERVICE_PATH = '/' -APIKEY = SERVICE_PATH.strip('/') +# FIWARE-Service path +SERVICE_PATH = '/' + +# ToDo: Change the APIKEY to something unique. This represents the "token" +# for IoT devices to connect (send/receive data) with the platform. In the +# context of MQTT, APIKEY is linked with the topic used for communication. +APIKEY = 'your_apikey' UNIQUE_ID = str(uuid4()) TOPIC_CONTROLLER = f"fiware_workshop/{UNIQUE_ID}/controller" print(TOPIC_CONTROLLER) -# Path to read json-files from previous exercises +# path to read json-files from previous exercises READ_GROUPS_FILEPATH = \ - Path("../e5_iot_thermal_zone_control_solution_groups.json") + Path("../e5_iot_thermal_zone_control_groups.json") READ_DEVICES_FILEPATH = \ - Path("../e5_iot_thermal_zone_control_solution_devices.json") + Path("../e5_iot_thermal_zone_control_devices.json") READ_SUBSCRIPTIONS_FILEPATH = \ - Path("../e5_iot_thermal_zone_control_solution_subscriptions.json") + Path("../e5_iot_thermal_zone_control_subscriptions.json") + +# opening the files +with open(READ_GROUPS_FILEPATH, 'r') as groups_file, \ + open(READ_DEVICES_FILEPATH, 'r') as devices_file, \ + open(READ_SUBSCRIPTIONS_FILEPATH, 'r') as subscriptions_file: + json_groups = json.load(groups_file) + json_devices = json.load(devices_file) + json_subscriptions = json.load(subscriptions_file) # set parameters for the temperature simulation TEMPERATURE_MAX = 10 # maximal ambient temperature @@ -87,7 +100,7 @@ T_SIM_START = 0 # simulation start time in seconds T_SIM_END = 24 * 60 * 60 # simulation end time in seconds -COM_STEP = 60 * 60 * 0.25 # 15 min communication step in seconds +COM_STEP = 60 * 60 * 0.25 # 15 min communication step in seconds # ## Main script if __name__ == '__main__': @@ -106,10 +119,10 @@ temp_min=TEMPERATURE_MIN, temp_start=TEMPERATURE_ZONE_START) - # Create clients and restore devices and groups from file - groups = parse_file_as(List[ServiceGroup], READ_GROUPS_FILEPATH) - devices = parse_file_as(List[Device], READ_DEVICES_FILEPATH) - sub = parse_file_as(List[Subscription], READ_SUBSCRIPTIONS_FILEPATH)[0] + # create clients and restore devices and groups from file + groups = TypeAdapter(List[ServiceGroup]).validate_python(json_groups) + devices = TypeAdapter(List[Device]).validate_python(json_devices) + sub = TypeAdapter(List[Subscription]).validate_python(json_subscriptions)[0] sub.notification.mqtt.topic = TOPIC_CONTROLLER sub.notification.mqtt.user = MQTT_USER sub.notification.mqtt.passwd = MQTT_PW @@ -119,24 +132,25 @@ iotac.post_groups(service_groups=groups) iotac.post_devices(devices=devices) - # Get the group and device configurations from the server + # get the group and device configurations from the server group = iotac.get_group(resource="/iot/json", apikey=APIKEY) weather_station = iotac.get_device(device_id="device:001") zone_temperature_sensor = iotac.get_device(device_id="device:002") heater = iotac.get_device(device_id="device:003") # create a MQTTv5 client with paho-mqtt and the known groups and - # devices. + # devices. mqttc = IoTAMQTTClient(protocol=mqtt.MQTTv5, devices=[weather_station, zone_temperature_sensor, heater], service_groups=[group]) - # ToDo: set user data if required + # ToDo: Set user data if required. mqttc.username_pw_set(username=MQTT_USER, password=MQTT_PW) # Implement a callback function that gets triggered when the - # command is sent to the device. The incoming command schould update the - # heater attribute of the simulation model + # command is sent to the device. The incoming command should update the + # heater attribute of the simulation model + def on_command(client, obj, msg): """ Callback for incoming commands @@ -148,33 +162,30 @@ def on_command(client, obj, msg): sim_model.heater_on = payload[heater.commands[0].name] - # acknowledge the command. Here command are usually single - # messages. The first key is equal to the commands name. + # Acknowledge the command. Here commands are usually single + # messages. The first key is equal to the commands name. client.publish(device_id=device_id, command_name=next(iter(payload)), payload=payload) # Add the command callback to your MQTTClient. This will get - # triggered for the specified device_id + # triggered for the specified device_id. mqttc.add_command_callback(device_id=heater.device_id, callback=on_command) # You need to implement a controller that controls the - # heater state with respect to the zone temperature. This will be - # implemented with asynchronous communication using MQTT-Subscriptions + # heater state with respect to the zone temperature. This will be + # implemented with asynchronous communication using MQTT-Subscriptions def on_measurement(client, obj, msg): """ Callback for measurement notifications """ - message = Message.parse_raw(msg.payload) + message = Message.model_validate_json(msg.payload) updated_zone_temperature_sensor = message.data[0] # retrieve the value of temperature attribute temperature = updated_zone_temperature_sensor.temperature.value - # device if you want update your command - # Note that this could also be substitute by a conditional - # subscription update = True if temperature <= 19: state = 1 @@ -192,21 +203,32 @@ def on_measurement(client, obj, msg): mqttc.message_callback_add(sub=TOPIC_CONTROLLER, callback=on_measurement) - # ToDo: create a quantumleap client - qlc = QuantumLeapClient(...) - - # ToDO: create a http subscriptions that get triggered by updates of your + # ToDo: Create http subscriptions that get triggered by updates of your # device attributes. Note that you can only post the subscription # to the context broker. - qlc.post_subscription(entity_id=weather_station.entity_name, - entity_type=weather_station.entity_type, - cb_url="http://orion:1026", - ql_url="http://quantumleap:8668", - throttling=0) + # Subscription for weather station + cbc.post_subscription(subscription=Subscription( + subject=Subject(**{ + 'entities': [{'id': weather_station.entity_name, + 'type': weather_station.entity_type}] + }), + notification=Notification(**{ + 'http': {'url': 'http://quantumleap:8668/v2/notify'} + }), + throttling=0) + ) - qlc.post_subscription(...) + # Subscription for zone temperature sensor + cbc.post_subscription(subscription=Subscription( + ... + ) + ) - qlc.post_subscription(...) + # Subscription for heater + cbc.post_subscription(subscription=Subscription( + ... + ) + ) # connect to the mqtt broker and subscribe to your topic mqtt_url = urlparse(MQTT_BROKER_URL_EXPOSED) @@ -217,6 +239,7 @@ def on_measurement(client, obj, msg): bind_port=0, clean_start=mqtt.MQTT_CLEAN_START_FIRST_ONLY, properties=None) + # subscribe to topics # subscribe to all incoming command topics for the registered devices mqttc.subscribe() @@ -225,32 +248,32 @@ def on_measurement(client, obj, msg): # create a non-blocking thread for mqtt communication mqttc.loop_start() - # Create a loop that publishes every second a message to the broker - # that holds the simulation time "simtime" and the corresponding - # temperature "temperature" the loop should. You may use the `object_id` - # or the attribute name as key in your payload. + # Create a loop that publishes a message every 0.3 seconds to the broker + # that holds the simulation time "sim_time" and the corresponding + # temperature "temperature". You may use the `object_id` + # or the attribute name as key in your payload. for t_sim in range(sim_model.t_start, sim_model.t_end + int(COM_STEP), int(COM_STEP)): # publish the simulated ambient temperature mqttc.publish(device_id=weather_station.device_id, payload={"temperature": sim_model.t_amb, - "simtime": sim_model.t_sim}) + "sim_time": sim_model.t_sim}) # publish the simulated zone temperature mqttc.publish(device_id=zone_temperature_sensor.device_id, payload={"temperature": sim_model.t_zone, - "simtime": sim_model.t_sim}) + "sim_time": sim_model.t_sim}) - # publish the 'simtime' for the heater device + # publish the 'sim_time' for the heater device mqttc.publish(device_id=heater.device_id, - payload={"simtime": sim_model.t_sim}) + payload={"sim_time": sim_model.t_sim}) - time.sleep(1) + time.sleep(0.3) # simulation step for next loop sim_model.do_step(int(t_sim + COM_STEP)) - # wait for one second before publishing the next values - time.sleep(1) + # wait for 0.3 seconds before publishing the next values + time.sleep(0.3) # close the mqtt listening thread mqttc.loop_stop() @@ -260,9 +283,12 @@ def on_measurement(client, obj, msg): # wait until all data is available time.sleep(10) + # Todo: Create a quantumleap client. + qlc = QuantumLeapClient(...) + # ToDo: Retrieve the historic data from QuantumLeap, convert them to a - # pandas dataframe and plot them - # Retrieve the data for the weather station + # pandas dataframe and plot them. + # retrieve the data for the weather station history_weather_station = qlc.get_entity_by_id( entity_id=weather_station.entity_name, entity_type=weather_station.entity_type, @@ -275,38 +301,40 @@ def on_measurement(client, obj, msg): # drop unnecessary index levels history_weather_station = history_weather_station.droplevel( level=("entityId", "entityType"), axis=1) - history_weather_station['simtime'] = pd.to_numeric( - history_weather_station['simtime'], downcast="float") + history_weather_station['sim_time'] = pd.to_numeric( + history_weather_station['sim_time'], downcast="float") history_weather_station['temperature'] = pd.to_numeric( history_weather_station['temperature'], downcast="float") - # ToDo: plot the results + # ToDo: Plot the results. fig, ax = plt.subplots() - ax.plot(history_weather_station['simtime'], + ax.plot(history_weather_station['sim_time']/60, history_weather_station['temperature']) - ax.set_xlabel('time in s') + ax.title.set_text("Weather Station") + ax.set_xlabel('time in min') ax.set_ylabel('ambient temperature in °C') plt.show() - # ToDo: Retrieve the data for the zone temperature + # ToDo: Retrieve the data for the zone temperature. history_zone_temperature_sensor = ... - # ToDo: convert to pandas dataframe and print it + # ToDo: Convert to pandas dataframe and print it. history_zone_temperature_sensor = ... - # ToDo: drop unnecessary index levels + # ToDo: Drop unnecessary index levels. history_zone_temperature_sensor = ... - history_zone_temperature_sensor['simtime'] = pd.to_numeric( - history_zone_temperature_sensor['simtime'], downcast="float") + history_zone_temperature_sensor['sim_time'] = pd.to_numeric( + history_zone_temperature_sensor['sim_time'], downcast="float") history_zone_temperature_sensor['temperature'] = pd.to_numeric( history_zone_temperature_sensor['temperature'], downcast="float") - # plot the results + # ToDo: Plot the results. fig2, ax2 = plt.subplots() - ax2.plot(history_zone_temperature_sensor['simtime'], + ax2.plot(history_zone_temperature_sensor['sim_time']/60, history_zone_temperature_sensor['temperature']) - ax2.set_xlabel('time in s') + ax2.title.set_text("Zone Temperature Sensor") + ax2.set_xlabel('time in min') ax2.set_ylabel('zone temperature in °C') plt.show() - # ToDo: Retrieve the data for the heater + # ToDo: Retrieve the data for the heater. history_heater = ... # convert to pandas dataframe and print it @@ -314,18 +342,19 @@ def on_measurement(client, obj, msg): history_heater = history_heater.replace(' ', 0) print(history_heater) - # drop unnecessary index levels + # ToDo: Drop unnecessary index levels. history_heater = history_heater.droplevel( level=("entityId", "entityType"), axis=1) - history_heater['simtime'] = pd.to_numeric( - history_heater['simtime'], downcast="float") + history_heater['sim_time'] = pd.to_numeric( + history_heater['sim_time'], downcast="float") history_heater['heater_on_info'] = pd.to_numeric( history_heater['heater_on_info'], downcast="float") - # plot the results + # ToDo: Plot the results. fig3, ax3 = plt.subplots() - ax3.plot(history_heater['simtime'], + ax3.plot(history_heater['sim_time']/60, history_heater['heater_on_info']) - ax3.set_xlabel('time in s') + ax3.title.set_text("Heater") + ax3.set_xlabel('time in min') ax3.set_ylabel('set point') plt.show() diff --git a/tutorials/ngsi_v2/e6_timeseries_data/e6_timeseries_data_solution.py b/tutorials/ngsi_v2/e6_timeseries_data/e6_timeseries_data_solution.py index 82cc4128..b1978407 100644 --- a/tutorials/ngsi_v2/e6_timeseries_data/e6_timeseries_data_solution.py +++ b/tutorials/ngsi_v2/e6_timeseries_data/e6_timeseries_data_solution.py @@ -7,13 +7,14 @@ # #### Steps to complete: # 1. Set up the missing parameters in the parameter section -# 2. Create a quantumleap client that create subscription that get triggered +# 2. Create a quantumleap client that creates subscription that gets triggered # by the updates on your context entities # 3. Run the simulation # 4. Retrieve the data via QuantumLeap and visualize it """ # ## Import packages +import json from pathlib import Path import time from typing import List @@ -21,7 +22,7 @@ from uuid import uuid4 import pandas as pd import paho.mqtt.client as mqtt -from pydantic import parse_file_as +from pydantic import TypeAdapter import matplotlib.pyplot as plt # import from filip from filip.clients.ngsi_v2 import \ @@ -31,7 +32,8 @@ from filip.clients.mqtt import IoTAMQTTClient from filip.models.base import FiwareHeader from filip.models.ngsi_v2.context import NamedCommand -from filip.models.ngsi_v2.subscriptions import Subscription, Message +from filip.models.ngsi_v2.subscriptions import Subscription, Message, Notification, \ + Subject from filip.models.ngsi_v2.iot import \ Device, \ PayloadProtocol, \ @@ -44,34 +46,38 @@ from tutorials.ngsi_v2.simulation_model import SimulationModel # ## Parameters -# ToDo: Enter your context broker host and port, e.g http://localhost:1026 +# ToDo: Enter your context broker host and port, e.g. http://localhost:1026. CB_URL = "http://localhost:1026" -# ToDo: Enter your IoT-Agent host and port, e.g http://localhost:4041 +# ToDo: Enter your IoT-Agent host and port, e.g. http://localhost:4041. IOTA_URL = "http://localhost:4041" -# ToDo: Enter your IoT-Agent host and port, e.g http://localhost:4041 +# ToDo: Enter your IoT-Agent host and port, e.g. http://localhost:4041. QL_URL = "http://localhost:8668" -# ToDo: Enter your mqtt broker url, e.g mqtt://test.mosquitto.org:1883 +# ToDo: Enter your mqtt broker url, e.g. mqtt://test.mosquitto.org:1883. MQTT_BROKER_URL_EXPOSED = "mqtt://localhost:1883" -# ToDo: Enter your mqtt broker url, e.g mqtt://mosquitto:1883 +# ToDo: Enter your mqtt broker url, e.g. mqtt://mosquitto:1883. MQTT_BROKER_URL_INTERNAL = "mqtt://mosquitto:1883" -# ToDo: If required enter your username and password +# ToDo: If required, enter your username and password. MQTT_USER = "" -MQTT_PW = "" +MQTT_PW = "" +# ToDo: Change the name of your service to something unique. If you run +# on a shared instance this is very important in order to avoid user +# collisions. You will use this service through the whole tutorial. +# If you forget to change it, an error will be raised! # FIWARE-Service SERVICE = 'filip_tutorial' -# FIWARE-Servicepath -# ToDo: Change the name of your service-path to something unique. If you run -# on a shared instance this very important in order to avoid user -# collisions. You will use this service path through the whole tutorial. -# If you forget to change it an error will be raised! -SERVICE_PATH = '/your_path' -APIKEY = SERVICE_PATH.strip('/') +# FIWARE-Service path +SERVICE_PATH = '/' + +# ToDo: Change the APIKEY to something unique. This represents the "token" +# for IoT devices to connect (send/receive data) with the platform. In the +# context of MQTT, APIKEY is linked with the topic used for communication. +APIKEY = 'your_apikey' UNIQUE_ID = str(uuid4()) TOPIC_CONTROLLER = f"fiware_workshop/{UNIQUE_ID}/controller" print(TOPIC_CONTROLLER) -# Path to read json-files from previous exercises +# path to read json-files from previous exercises READ_GROUPS_FILEPATH = \ Path("../e5_iot_thermal_zone_control_solution_groups.json") READ_DEVICES_FILEPATH = \ @@ -79,6 +85,14 @@ READ_SUBSCRIPTIONS_FILEPATH = \ Path("../e5_iot_thermal_zone_control_solution_subscriptions.json") +# opening the files +with open(READ_GROUPS_FILEPATH, 'r') as groups_file, \ + open(READ_DEVICES_FILEPATH, 'r') as devices_file, \ + open(READ_SUBSCRIPTIONS_FILEPATH, 'r') as subscriptions_file: + json_groups = json.load(groups_file) + json_devices = json.load(devices_file) + json_subscriptions = json.load(subscriptions_file) + # set parameters for the temperature simulation TEMPERATURE_MAX = 10 # maximal ambient temperature TEMPERATURE_MIN = -5 # minimal ambient temperature @@ -86,7 +100,7 @@ T_SIM_START = 0 # simulation start time in seconds T_SIM_END = 24 * 60 * 60 # simulation end time in seconds -COM_STEP = 60 * 60 * 0.25 # 15 min communication step in seconds +COM_STEP = 60 * 60 * 0.25 # 15 min communication step in seconds # ## Main script if __name__ == '__main__': @@ -105,10 +119,10 @@ temp_min=TEMPERATURE_MIN, temp_start=TEMPERATURE_ZONE_START) - # Create clients and restore devices and groups from file - groups = parse_file_as(List[ServiceGroup], READ_GROUPS_FILEPATH) - devices = parse_file_as(List[Device], READ_DEVICES_FILEPATH) - sub = parse_file_as(List[Subscription], READ_SUBSCRIPTIONS_FILEPATH)[0] + # create clients and restore devices and groups from file + groups = TypeAdapter(List[ServiceGroup]).validate_python(json_groups) + devices = TypeAdapter(List[Device]).validate_python(json_devices) + sub = TypeAdapter(List[Subscription]).validate_python(json_subscriptions)[0] sub.notification.mqtt.topic = TOPIC_CONTROLLER sub.notification.mqtt.user = MQTT_USER sub.notification.mqtt.passwd = MQTT_PW @@ -118,24 +132,25 @@ iotac.post_groups(service_groups=groups) iotac.post_devices(devices=devices) - # Get the group and device configurations from the server + # get the group and device configurations from the server group = iotac.get_group(resource="/iot/json", apikey=APIKEY) weather_station = iotac.get_device(device_id="device:001") zone_temperature_sensor = iotac.get_device(device_id="device:002") heater = iotac.get_device(device_id="device:003") # create a MQTTv5 client with paho-mqtt and the known groups and - # devices. + # devices. mqttc = IoTAMQTTClient(protocol=mqtt.MQTTv5, devices=[weather_station, zone_temperature_sensor, heater], service_groups=[group]) - # ToDo: set user data if required + # ToDo: Set user data if required. mqttc.username_pw_set(username=MQTT_USER, password=MQTT_PW) # Implement a callback function that gets triggered when the - # command is sent to the device. The incoming command schould update the - # heater attribute of the simulation model + # command is sent to the device. The incoming command should update the + # heater attribute of the simulation model + def on_command(client, obj, msg): """ Callback for incoming commands @@ -147,33 +162,30 @@ def on_command(client, obj, msg): sim_model.heater_on = payload[heater.commands[0].name] - # acknowledge the command. Here command are usually single - # messages. The first key is equal to the commands name. + # Acknowledge the command. Here commands are usually single + # messages. The first key is equal to the commands name. client.publish(device_id=device_id, command_name=next(iter(payload)), payload=payload) # Add the command callback to your MQTTClient. This will get - # triggered for the specified device_id + # triggered for the specified device_id. mqttc.add_command_callback(device_id=heater.device_id, callback=on_command) # You need to implement a controller that controls the - # heater state with respect to the zone temperature. This will be - # implemented with asynchronous communication using MQTT-Subscriptions + # heater state with respect to the zone temperature. This will be + # implemented with asynchronous communication using MQTT-Subscriptions def on_measurement(client, obj, msg): """ Callback for measurement notifications """ - message = Message.parse_raw(msg.payload) + message = Message.model_validate_json(msg.payload) updated_zone_temperature_sensor = message.data[0] # retrieve the value of temperature attribute temperature = updated_zone_temperature_sensor.temperature.value - # device if you want update your command - # Note that this could also be substitute by a conditional - # subscription update = True if temperature <= 19: state = 1 @@ -191,29 +203,42 @@ def on_measurement(client, obj, msg): mqttc.message_callback_add(sub=TOPIC_CONTROLLER, callback=on_measurement) - # ToDo: create a quantumleap client - qlc = QuantumLeapClient(url=QL_URL, fiware_header=fiware_header) - - # ToDO: create a http subscriptions that get triggered by updates of your + # ToDo: Create http subscriptions that get triggered by updates of your # device attributes. Note that you can only post the subscription # to the context broker. - qlc.post_subscription(entity_id=weather_station.entity_name, - entity_type=weather_station.entity_type, - cb_url="http://orion:1026", - ql_url="http://quantumleap:8668", - throttling=0) - - qlc.post_subscription(entity_id=zone_temperature_sensor.entity_name, - entity_type=zone_temperature_sensor.entity_type, - cb_url="http://orion:1026", - ql_url="http://quantumleap:8668", - throttling=0) - - qlc.post_subscription(entity_id=heater.entity_name, - entity_type=heater.entity_type, - cb_url="http://orion:1026", - ql_url="http://quantumleap:8668", - throttling=0) + + cbc.post_subscription(subscription=Subscription( + subject=Subject(**{ + 'entities': [{'id': weather_station.entity_name, + 'type': weather_station.entity_type}] + }), + notification=Notification(**{ + 'http': {'url': 'http://quantumleap:8668/v2/notify'} + }), + throttling=0) + ) + + cbc.post_subscription(subscription=Subscription( + subject=Subject(**{ + 'entities': [{'id': zone_temperature_sensor.entity_name, + 'type': zone_temperature_sensor.entity_type}] + }), + notification=Notification(**{ + 'http': {'url': 'http://quantumleap:8668/v2/notify'} + }), + throttling=0) + ) + + cbc.post_subscription(subscription=Subscription( + subject=Subject(**{ + 'entities': [{'id': heater.entity_name, + 'type': heater.entity_type}] + }), + notification=Notification(**{ + 'http': {'url': 'http://quantumleap:8668/v2/notify'} + }), + throttling=0) + ) # connect to the mqtt broker and subscribe to your topic mqtt_url = urlparse(MQTT_BROKER_URL_EXPOSED) @@ -224,6 +249,7 @@ def on_measurement(client, obj, msg): bind_port=0, clean_start=mqtt.MQTT_CLEAN_START_FIRST_ONLY, properties=None) + # subscribe to topics # subscribe to all incoming command topics for the registered devices mqttc.subscribe() @@ -232,32 +258,32 @@ def on_measurement(client, obj, msg): # create a non-blocking thread for mqtt communication mqttc.loop_start() - # Create a loop that publishes every second a message to the broker - # that holds the simulation time "simtime" and the corresponding - # temperature "temperature" the loop should. You may use the `object_id` - # or the attribute name as key in your payload. + # Create a loop that publishes a message every 0.3 seconds to the broker + # that holds the simulation time "sim_time" and the corresponding + # temperature "temperature". You may use the `object_id` + # or the attribute name as key in your payload. for t_sim in range(sim_model.t_start, sim_model.t_end + int(COM_STEP), int(COM_STEP)): # publish the simulated ambient temperature mqttc.publish(device_id=weather_station.device_id, payload={"temperature": sim_model.t_amb, - "simtime": sim_model.t_sim}) + "sim_time": sim_model.t_sim}) # publish the simulated zone temperature mqttc.publish(device_id=zone_temperature_sensor.device_id, payload={"temperature": sim_model.t_zone, - "simtime": sim_model.t_sim}) + "sim_time": sim_model.t_sim}) - # publish the 'simtime' for the heater device + # publish the 'sim_time' for the heater device mqttc.publish(device_id=heater.device_id, - payload={"simtime": sim_model.t_sim}) + payload={"sim_time": sim_model.t_sim}) - time.sleep(1) + time.sleep(0.3) # simulation step for next loop sim_model.do_step(int(t_sim + COM_STEP)) - # wait for one second before publishing the next values - time.sleep(1) + # wait for 0.3 seconds before publishing the next values + time.sleep(0.3) # close the mqtt listening thread mqttc.loop_stop() @@ -267,9 +293,12 @@ def on_measurement(client, obj, msg): # wait until all data is available time.sleep(10) + # ToDo: Create a quantumleap client. + qlc = QuantumLeapClient(url=QL_URL, fiware_header=fiware_header) + # ToDo: Retrieve the historic data from QuantumLeap, convert them to a - # pandas dataframe and plot them - # Retrieve the data for the weather station + # pandas dataframe and plot them. + # retrieve the data for the weather station history_weather_station = qlc.get_entity_by_id( entity_id=weather_station.entity_name, entity_type=weather_station.entity_type, @@ -282,45 +311,47 @@ def on_measurement(client, obj, msg): # drop unnecessary index levels history_weather_station = history_weather_station.droplevel( level=("entityId", "entityType"), axis=1) - history_weather_station['simtime'] = pd.to_numeric( - history_weather_station['simtime'], downcast="float") + history_weather_station['sim_time'] = pd.to_numeric( + history_weather_station['sim_time'], downcast="float") history_weather_station['temperature'] = pd.to_numeric( history_weather_station['temperature'], downcast="float") - # ToDo: plot the results + # ToDo: Plot the results. fig, ax = plt.subplots() - ax.plot(history_weather_station['simtime'], + ax.plot(history_weather_station['sim_time']/60, history_weather_station['temperature']) - ax.set_xlabel('time in s') + ax.title.set_text("Weather Station") + ax.set_xlabel('time in min') ax.set_ylabel('ambient temperature in °C') plt.show() - # ToDo: Retrieve the data for the zone temperature + # ToDo: Retrieve the data for the zone temperature. history_zone_temperature_sensor = qlc.get_entity_by_id( entity_id=zone_temperature_sensor.entity_name, entity_type=zone_temperature_sensor.entity_type, last_n=10000 ) - # ToDo: convert to pandas dataframe and print it + # ToDo: Convert to pandas dataframe and print it. history_zone_temperature_sensor = \ history_zone_temperature_sensor.to_pandas() print(history_zone_temperature_sensor) - # ToDo: drop unnecessary index levels + # ToDo: Drop unnecessary index levels. history_zone_temperature_sensor = history_zone_temperature_sensor.droplevel( level=("entityId", "entityType"), axis=1) - history_zone_temperature_sensor['simtime'] = pd.to_numeric( - history_zone_temperature_sensor['simtime'], downcast="float") + history_zone_temperature_sensor['sim_time'] = pd.to_numeric( + history_zone_temperature_sensor['sim_time'], downcast="float") history_zone_temperature_sensor['temperature'] = pd.to_numeric( history_zone_temperature_sensor['temperature'], downcast="float") - # ToDo: plot the results + # ToDo: Plot the results. fig2, ax2 = plt.subplots() - ax2.plot(history_zone_temperature_sensor['simtime'], + ax2.plot(history_zone_temperature_sensor['sim_time']/60, history_zone_temperature_sensor['temperature']) - ax2.set_xlabel('time in s') + ax2.title.set_text("Zone Temperature Sensor") + ax2.set_xlabel('time in min') ax2.set_ylabel('zone temperature in °C') plt.show() - # ToDo: Retrieve the data for the heater + # ToDo: Retrieve the data for the heater. history_heater = qlc.get_entity_by_id( entity_id=heater.entity_name, entity_type=heater.entity_type, @@ -332,18 +363,19 @@ def on_measurement(client, obj, msg): history_heater = history_heater.replace(' ', 0) print(history_heater) - # ToDo: drop unnecessary index levels + # ToDo: Drop unnecessary index levels. history_heater = history_heater.droplevel( level=("entityId", "entityType"), axis=1) - history_heater['simtime'] = pd.to_numeric( - history_heater['simtime'], downcast="float") + history_heater['sim_time'] = pd.to_numeric( + history_heater['sim_time'], downcast="float") history_heater['heater_on_info'] = pd.to_numeric( history_heater['heater_on_info'], downcast="float") - # ToDo: plot the results + # ToDo: Plot the results. fig3, ax3 = plt.subplots() - ax3.plot(history_heater['simtime'], + ax3.plot(history_heater['sim_time']/60, history_heater['heater_on_info']) - ax3.set_xlabel('time in s') + ax3.title.set_text("Heater") + ax3.set_xlabel('time in min') ax3.set_ylabel('set point') plt.show() diff --git a/tutorials/ngsi_v2/e7_semantic_iot/e7_semantic_iot.py b/tutorials/ngsi_v2/e7_semantic_iot/e7_semantic_iot.py index d166e965..5c536a1f 100644 --- a/tutorials/ngsi_v2/e7_semantic_iot/e7_semantic_iot.py +++ b/tutorials/ngsi_v2/e7_semantic_iot/e7_semantic_iot.py @@ -2,7 +2,7 @@ # # Exercise 7: Semantic IoT Systems # # We now want to add a semantic meaning to our measurements. Therefore we -# semantically connect to the context entities that we created in +# semantically connect the context entities that we created in # `e3_context_entities.py` # The input sections are marked with 'ToDo' @@ -14,14 +14,15 @@ # 3. Add relationships that connect the thermal zone temperature sensor and # the heater to the building, and vice versa # 4. Retrieve all entities and print them -# 5. Congratulations you are now ready to build your own semantic systems for +# 5. Congratulations! You are now ready to build your own semantic systems. For # advanced semantic functions check on our semantics examples """ # ## Import packages +import json from pathlib import Path from typing import List -from pydantic import parse_file_as +from pydantic import TypeAdapter # import from filip from filip.clients.ngsi_v2 import \ @@ -42,28 +43,40 @@ clear_iot_agent # ## Parameters -# ToDo: Enter your context broker host and port, e.g http://localhost:1026 +# ToDo: Enter your context broker host and port, e.g. http://localhost:1026. CB_URL = "http://localhost:1026" -# ToDo: Enter your IoT-Agent host and port, e.g http://localhost:4041 +# ToDo: Enter your IoT-Agent host and port, e.g. http://localhost:4041. IOTA_URL = "http://localhost:4041" # FIWARE-Service SERVICE = 'filip_tutorial' -# FIWARE-Servicepath +# FIWARE-Service path # ToDo: Change the name of your service-path to something unique. If you run -# on a shared instance this very important in order to avoid user +# on a shared instance this is very important in order to avoid user # collisions. You will use this service path through the whole tutorial. -# If you forget to change it an error will be raised! -SERVICE_PATH = '/' -APIKEY = SERVICE_PATH.strip('/') +# If you forget to change it, an error will be raised! +SERVICE_PATH = '/' -# Path to read json-files from previous exercises +# ToDo: Change the APIKEY to something unique. This represents the "token" +# for IoT devices to connect (send/receive data) with the platform. In the +# context of MQTT, APIKEY is linked with the topic used for communication. +APIKEY = 'your_apikey' + +# path to read json-files from previous exercises READ_GROUPS_FILEPATH = \ Path("../e5_iot_thermal_zone_control_groups.json") READ_DEVICES_FILEPATH = \ Path("../e5_iot_thermal_zone_control_devices.json") READ_ENTITIES_FILEPATH = \ - Path("../e3_context_entities_entities.json") + Path("../e3_context_entities.json") + +# opening the files +with open(READ_GROUPS_FILEPATH, 'r') as groups_file, \ + open(READ_DEVICES_FILEPATH, 'r') as devices_file, \ + open(READ_ENTITIES_FILEPATH, 'r') as entities_file: + json_groups = json.load(groups_file) + json_devices = json.load(devices_file) + json_entities = json.load(entities_file) # ## Main script if __name__ == '__main__': @@ -74,10 +87,10 @@ clear_iot_agent(url=IOTA_URL, fiware_header=fiware_header) clear_context_broker(url=CB_URL, fiware_header=fiware_header) - # Create clients and restore devices and groups from file - groups = parse_file_as(List[ServiceGroup], READ_GROUPS_FILEPATH) - devices = parse_file_as(List[Device], READ_DEVICES_FILEPATH) - entities = parse_file_as(List[ContextEntity], READ_ENTITIES_FILEPATH) + # create clients and restore devices and groups from file + groups = TypeAdapter(List[ServiceGroup]).validate_python(json_groups) + devices = TypeAdapter(List[Device]).validate_python(json_devices) + entities = TypeAdapter(List[ContextEntity]).validate_python(json_entities) cbc = ContextBrokerClient(url=CB_URL, fiware_header=fiware_header) for entity in entities: cbc.post_entity(entity=entity) @@ -86,15 +99,15 @@ iotac.post_groups(service_groups=groups) iotac.post_devices(devices=devices) - # ToDo: Retrieve all iot resources from the IoT-Agent - # Get the group and device configurations from the server + # ToDo: Retrieve all iot resources from the IoT-Agent. + # get the group and device configurations from the server group = iotac.get_group(resource="/iot/json", apikey=APIKEY) weather_station = iotac.get_device(device_id="device:001") zone_temperature_sensor = iotac.get_device(device_id="device:002") heater = iotac.get_device(device_id="device:003") # ToDo: Get context entities from the Context Broker - # (exclude the IoT device ones) + # (exclude the IoT device ones). building = cbc.get_entity(entity_id="urn:ngsi-ld:building:001", entity_type="Building") thermal_zone = cbc.get_entity(entity_id="ThermalZone:001", @@ -103,7 +116,7 @@ # ToDo: Semantically connect the weather station and the building. By # adding a `hasWeatherStation` attribute of type `Relationship`. For the # connection from the weather station to the building add a static - # attribute to the weather station + # attribute to the weather station. # create the context attribute for the building and add it to the # building entity @@ -127,23 +140,22 @@ # zone by adding a `hasTemperatureSensor` attribute of type # `Relationship` to the thermal zone entity. # For the connection from the sensor to the zone add a static - # attribute to the temperature sensor device + # attribute to the temperature sensor device. - # ToDo: create the context attribute for the thermal zone and add it to the - # thermal zone entity - has_sensor = ... + # ToDo: Create a context attribute for the thermal zone and add it to the + # thermal zone entity. + has_sensor = ... thermal_zone.add_attributes(...) - # ToDo: create a static attribute that connects the zone temperature zone to - # the thermal zone + # ToDo: Create a static attribute that connects the zone temperature zone to + # the thermal zone. cbc.update_entity(entity=thermal_zone) ref_thermal_zone = StaticDeviceAttribute(...) - zone_temperature_sensor.add_attribute(ref_thermal_zone) iotac.update_device(device=zone_temperature_sensor) @@ -151,18 +163,18 @@ # zone by adding a `hasTemperatureSensor` attribute of type # `Relationship` to the thermal zone entity. # For the connection from the sensor to the zone add a static - # attribute to the temperature sensor device + # attribute to the temperature sensor device. - # ToDo: create the context attribute for the thermal zone and add it to the - # thermal zone entity + # ToDo: Create a context attribute for the thermal zone and add it to the + # thermal zone entity. has_heater = ... thermal_zone.add_attributes(...) - # ToDo: create a static attribute that connects the zone temperature zone to - # the thermal zone + # ToDo: Create a static attribute that connects the zone temperature zone to + # the thermal zone. cbc.update_entity(entity=thermal_zone) ref_thermal_zone = ... @@ -171,27 +183,27 @@ heater.add_attribute(ref_thermal_zone) iotac.update_device(device=heater) - # ToDo: Add unit metadata to the temperature and simtime attributes of - # all devices. Here we use unitcode information. If you can not find - # your unit code you can use our unit models for help + # ToDo: Add unit metadata to the temperature and sim_time attributes of + # all devices. Here we use unit code information. If you can not find + # your unit code, you can use our unit models for help. # get code from Unit model for seconds code = Unit(name="second [unit of time]").code - # add metadata to simtime attribute of the all devices - metadata_simtime = NamedMetadata(name="unitCode", - type="Text", - value=code) - attr_simtime = weather_station.get_attribute( - attribute_name="simtime" + # add metadata to sim_time attribute of the all devices + metadata_sim_time = NamedMetadata(name="unitCode", + type="Text", + value=code) + attr_sim_time = weather_station.get_attribute( + attribute_name="sim_time" ) - attr_simtime.metadata = metadata_simtime - weather_station.update_attribute(attribute=attr_simtime) - zone_temperature_sensor.update_attribute(attribute=attr_simtime) - heater.update_attribute(attribute=attr_simtime) + attr_sim_time.metadata = metadata_sim_time + weather_station.update_attribute(attribute=attr_sim_time) + zone_temperature_sensor.update_attribute(attribute=attr_sim_time) + heater.update_attribute(attribute=attr_sim_time) - # ToDo: get code from Unit model for degree celsius + # ToDo: Get code from Unit model for degree celsius. code = Unit(name="degree Celsius").code - # ToDo: add metadata to temperature attribute of the weather - # station and the zone temperature sensor + # ToDo: Add metadata to temperature attribute of the weather + # station and the zone temperature sensor. metadata_t_amb = NamedMetadata(...) attr_t_amb = weather_station.get_attribute( attribute_name="temperature" @@ -205,8 +217,8 @@ attr_t_zone.metadata = ... zone_temperature_sensor.update_attribute(...) - # currently adding metadata via updating does not work perfectly. - # Therefore, we do a delete and update + # Currently adding metadata via updating does not work perfectly, + # therefore, we delete and update them. iotac.delete_device(device_id=weather_station.device_id) iotac.post_device(device=weather_station) iotac.delete_device(device_id=zone_temperature_sensor.device_id) @@ -214,10 +226,10 @@ iotac.delete_device(device_id=heater.device_id) iotac.post_device(device=heater) - # ToDo: Retrieve all ContextEntites and print them + # ToDo: Retrieve all Context Entities and print them. entities = ... for entity in entities: - print(entity.json(indent=2)) + print(entity.model_dump_json(indent=2)) clear_iot_agent(url=IOTA_URL, fiware_header=fiware_header) clear_context_broker(url=CB_URL, fiware_header=fiware_header) diff --git a/tutorials/ngsi_v2/e7_semantic_iot/e7_semantic_iot_solutions.py b/tutorials/ngsi_v2/e7_semantic_iot/e7_semantic_iot_solutions.py index 1256e30a..680a81ac 100644 --- a/tutorials/ngsi_v2/e7_semantic_iot/e7_semantic_iot_solutions.py +++ b/tutorials/ngsi_v2/e7_semantic_iot/e7_semantic_iot_solutions.py @@ -2,7 +2,7 @@ # # Exercise 7: Semantic IoT Systems # # We now want to add a semantic meaning to our measurements. Therefore we -# semantically connect to the context entities that we created in +# semantically connect the context entities that we created in # `e3_context_entities.py` # The input sections are marked with 'ToDo' @@ -14,14 +14,15 @@ # 3. Add relationships that connect the thermal zone temperature sensor and # the heater to the building, and vice versa # 4. Retrieve all entities and print them -# 5. Congratulations you are now ready to build your own semantic systems for +# 5. Congratulations! You are now ready to build your own semantic systems. For # advanced semantic functions check on our semantics examples """ # ## Import packages +import json from pathlib import Path from typing import List -from pydantic import parse_file_as +from pydantic import TypeAdapter # import from filip from filip.clients.ngsi_v2 import \ @@ -42,22 +43,26 @@ clear_iot_agent # ## Parameters -# ToDo: Enter your context broker host and port, e.g http://localhost:1026 +# ToDo: Enter your context broker host and port, e.g. http://localhost:1026. CB_URL = "http://localhost:1026" -# ToDo: Enter your IoT-Agent host and port, e.g http://localhost:4041 +# ToDo: Enter your IoT-Agent host and port, e.g. http://localhost:4041. IOTA_URL = "http://localhost:4041" +# ToDo: Change the name of your service to something unique. If you run +# on a shared instance this very is important in order to avoid user +# collisions. You will use this service through the whole tutorial. +# If you forget to change it, an error will be raised! # FIWARE-Service SERVICE = 'filip_tutorial' -# FIWARE-Servicepath -# ToDo: Change the name of your service-path to something unique. If you run -# on a shared instance this very important in order to avoid user -# collisions. You will use this service path through the whole tutorial. -# If you forget to change it an error will be raised! -SERVICE_PATH = '/your_path' -APIKEY = SERVICE_PATH.strip('/') - -# Path to read json-files from previous exercises +# FIWARE-Service path +SERVICE_PATH = '/' + +# ToDo: Change the APIKEY to something unique. This represents the "token" +# for IoT devices to connect (send/receive data) with the platform. In the +# context of MQTT, APIKEY is linked with the topic used for communication. +APIKEY = 'your_apikey' + +# path to read json-files from previous exercises READ_GROUPS_FILEPATH = \ Path("../e5_iot_thermal_zone_control_solution_groups.json") READ_DEVICES_FILEPATH = \ @@ -65,6 +70,14 @@ READ_ENTITIES_FILEPATH = \ Path("../e3_context_entities_solution_entities.json") +# opening the files +with open(READ_GROUPS_FILEPATH, 'r') as groups_file, \ + open(READ_DEVICES_FILEPATH, 'r') as devices_file, \ + open(READ_ENTITIES_FILEPATH, 'r') as entities_file: + json_groups = json.load(groups_file) + json_devices = json.load(devices_file) + json_entities = json.load(entities_file) + # ## Main script if __name__ == '__main__': # create a fiware header object @@ -74,10 +87,10 @@ clear_iot_agent(url=IOTA_URL, fiware_header=fiware_header) clear_context_broker(url=CB_URL, fiware_header=fiware_header) - # Create clients and restore devices and groups from file - groups = parse_file_as(List[ServiceGroup], READ_GROUPS_FILEPATH) - devices = parse_file_as(List[Device], READ_DEVICES_FILEPATH) - entities = parse_file_as(List[ContextEntity], READ_ENTITIES_FILEPATH) + # create clients and restore devices and groups from file + groups = TypeAdapter(List[ServiceGroup]).validate_python(json_groups) + devices = TypeAdapter(List[Device]).validate_python(json_devices) + entities = TypeAdapter(List[ContextEntity]).validate_python(json_entities) cbc = ContextBrokerClient(url=CB_URL, fiware_header=fiware_header) for entity in entities: cbc.post_entity(entity=entity) @@ -86,15 +99,15 @@ iotac.post_groups(service_groups=groups) iotac.post_devices(devices=devices) - # ToDo: Retrieve all iot resources from the IoT-Agent - # Get the group and device configurations from the server + # ToDo: Retrieve all iot resources from the IoT-Agent. + # get the group and device configurations from the server group = iotac.get_group(resource="/iot/json", apikey=APIKEY) weather_station = iotac.get_device(device_id="device:001") zone_temperature_sensor = iotac.get_device(device_id="device:002") heater = iotac.get_device(device_id="device:003") # ToDo: Get context entities from the Context Broker - # (exclude the IoT device ones) + # (exclude the IoT device ones). building = cbc.get_entity(entity_id="urn:ngsi-ld:building:001", entity_type="Building") thermal_zone = cbc.get_entity(entity_id="ThermalZone:001", @@ -127,18 +140,18 @@ # zone by adding a `hasTemperatureSensor` attribute of type # `Relationship` to the thermal zone entity. # For the connection from the sensor to the zone add a static - # attribute to the temperature sensor device + # attribute to the temperature sensor device. - # ToDo: create the context attribute for the thermal zone and add it to the - # thermal zone entity + # ToDo: Create a context attribute for the thermal zone and add it to the + # thermal zone entity. has_sensor = NamedContextAttribute( name="hasTemperatureSensor", type="Relationship", value=zone_temperature_sensor.entity_name) thermal_zone.add_attributes(attrs=[has_sensor]) - # ToDo: create a static attribute that connects the zone temperature zone to - # the thermal zone + # ToDo: Create a static attribute that connects the zone temperature zone to + # the thermal zone. cbc.update_entity(entity=thermal_zone) ref_thermal_zone = StaticDeviceAttribute(name="refThermalZone", @@ -151,18 +164,18 @@ # zone by adding a `hasTemperatureSensor` attribute of type # `Relationship` to the thermal zone entity. # For the connection from the sensor to the zone add a static - # attribute to the temperature sensor device + # attribute to the temperature sensor device. - # ToDo: create the context attribute for the thermal zone and add it to the - # thermal zone entity + # ToDo: Create a context attribute for the thermal zone and add it to the + # thermal zone entity. has_heater = NamedContextAttribute( name="hasHeater", type="Relationship", value=heater.entity_name) thermal_zone.add_attributes(attrs=[has_heater]) - # ToDo: create a static attribute that connects the zone temperature zone to - # the thermal zone + # ToDo: Create a static attribute that connects the zone temperature zone to + # the thermal zone. cbc.update_entity(entity=thermal_zone) ref_thermal_zone = StaticDeviceAttribute(name="refThermalZone", @@ -171,27 +184,27 @@ heater.add_attribute(ref_thermal_zone) iotac.update_device(device=heater) - # ToDo: Add unit metadata to the temperature and simtime attributes of - # all devices. Here we use unitcode information. If you can not find - # your unit code you can use our unit models for help + # ToDo: Add unit metadata to the temperature and sim_time attributes of + # all devices. Here we use unit code information. If you can not find + # your unit code, you can use our unit models for help. # get code from Unit model for seconds code = Unit(name="second [unit of time]").code - # add metadata to simtime attribute of the all devices - metadata_simtime = NamedMetadata(name="unitCode", - type="Text", - value=code) - attr_simtime = weather_station.get_attribute( - attribute_name="simtime" + # add metadata to sim_time attribute of the all devices + metadata_sim_time = NamedMetadata(name="unitCode", + type="Text", + value=code) + attr_sim_time = weather_station.get_attribute( + attribute_name="sim_time" ) - attr_simtime.metadata = metadata_simtime - weather_station.update_attribute(attribute=attr_simtime) - zone_temperature_sensor.update_attribute(attribute=attr_simtime) - heater.update_attribute(attribute=attr_simtime) + attr_sim_time.metadata = metadata_sim_time + weather_station.update_attribute(attribute=attr_sim_time) + zone_temperature_sensor.update_attribute(attribute=attr_sim_time) + heater.update_attribute(attribute=attr_sim_time) - # ToDo: get code from Unit model for degree celsius + # ToDo: Get code from Unit model for degree celsius. code = Unit(name="degree Celsius").code - # ToDo: add metadata to temperature attribute of the weather - # station and the zone temperature sensor + # ToDo: Add metadata to temperature attribute of the weather + # station and the zone temperature sensor. metadata_t_amb = NamedMetadata(name="unitCode", type="Text", value=code) @@ -202,15 +215,15 @@ weather_station.update_attribute(attribute=attr_t_amb) metadata_t_zone = NamedMetadata(name="unitCode", - type="Text", - value=code) + type="Text", + value=code) attr_t_zone = zone_temperature_sensor.get_attribute( attribute_name="temperature") attr_t_zone.metadata = metadata_t_zone zone_temperature_sensor.update_attribute(attribute=attr_t_zone) - # currently adding metadata via updating does not work perfectly. - # Therefore, we do a delete and update + # Currently adding metadata via updating does not work perfectly, + # therefore, we delete and update them. iotac.delete_device(device_id=weather_station.device_id) iotac.post_device(device=weather_station) iotac.delete_device(device_id=zone_temperature_sensor.device_id) @@ -218,10 +231,10 @@ iotac.delete_device(device_id=heater.device_id) iotac.post_device(device=heater) - # ToDo: Retrieve all ContextEntites and print them + # ToDo: Retrieve all Context Entities and print them. entities = cbc.get_entity_list() for entity in entities: - print(entity.json(indent=2)) + print(entity.model_dump_json(indent=2)) clear_iot_agent(url=IOTA_URL, fiware_header=fiware_header) clear_context_broker(url=CB_URL, fiware_header=fiware_header) diff --git a/tutorials/ngsi_v2/e8_multientity_and_expression_language/README.md b/tutorials/ngsi_v2/e8_multientity_and_expression_language/README.md new file mode 100644 index 00000000..11c27434 --- /dev/null +++ b/tutorials/ngsi_v2/e8_multientity_and_expression_language/README.md @@ -0,0 +1,21 @@ +# Exercise 8: MultiEntity and Expression Language + +The MultiEntity plugin Allows the devices provisioned in the IoTAgent to map their attributes to more than one entity, +declaring the target entity through the Configuration or Device provisioning APIs. + +The IoTAgent Library provides an expression language for measurement transformation, that can be used to adapt the +information coming from the South Bound APIs to the information reported to the Context Broker. This is really useful +when you need to adapt measured data. + +There are two different expression languages available, `jexl` and `legacy`. The recommended language is `jexl`, +which is newer and most powerful. + +We want to apply these features to `e8_multientity_and_expression_language.py`. + +The input sections are marked with 'TODO' + +#### Steps to complete: +1. Setting up the expression language jexl +2. Applying the expression language to device attributes +3. Testing the expression language via MQTT messages +4. Applying the expression language to device attributes in a multi-entity scenario \ No newline at end of file diff --git a/tutorials/ngsi_v2/e8_multientity_and_expression_language/__init__.py b/tutorials/ngsi_v2/e8_multientity_and_expression_language/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tutorials/ngsi_v2/e8_multientity_and_expression_language/e8_multientity_and_expression_language.py b/tutorials/ngsi_v2/e8_multientity_and_expression_language/e8_multientity_and_expression_language.py new file mode 100644 index 00000000..9e2b6966 --- /dev/null +++ b/tutorials/ngsi_v2/e8_multientity_and_expression_language/e8_multientity_and_expression_language.py @@ -0,0 +1,165 @@ +""" +# # Exercise 8: MultiEntity and Expression Language + +# The MultiEntity plugin allows the devices provisioned in the IoTAgent to map their +attributes to more than one entity, declaring the target entity through the +Configuration or Device provisioning APIs. + +# The IoTAgent Library provides an expression language for measurement transformation, +that can be used to adapt the # information coming from the South Bound APIs to the +information reported to the Context Broker. This is really useful when you need to +adapt measure. + +# There are available two different expression languages jexl and legacy. The +recommended language to use is jexl, which is newer and most powerful. + +# The input sections are marked with 'TODO' + +# #### Steps to complete: +# 1. Setting up the expression language jexl +# 2. Applying the expression language to device attributes +# 3. Testing the expression language via MQTT messages +# 4. Applying the expression language to device attributes in a multi-entity scenario +""" +# Import packages +import time +import datetime + +from filip.clients.ngsi_v2 import IoTAClient, ContextBrokerClient +from filip.models.base import FiwareHeader +from filip.models.ngsi_v2.context import ContextEntity, NamedContextAttribute +from filip.models.ngsi_v2.iot import (Device, ServiceGroup, TransportProtocol, + PayloadProtocol, DeviceAttribute, + ExpressionLanguage) +from filip.utils.cleanup import clear_all +from paho.mqtt import client as mqtt_client + +# Host address of Context Broker +CB_URL = "http://localhost:1026" + +# Host address of IoT-Agent +IOTA_URL = "http://localhost:4041" + +# MQTT Broker +MQTT_BROKER_HOST = "localhost" +MQTT_BROKER_PORT = 1883 + +# ToDo: Change the name of your service to something unique. If you run +# on a shared instance this very important in order to avoid user +# collisions. You will use this service through the whole tutorial. +# If you forget to change it an error will be raised! +# FIWARE Service +SERVICE = 'filip_tutorial' +SERVICE_PATH = '/' + +# ToDo: Change the APIKEY to something unique. This represent the "token" +# for IoT devices to connect (send/receive data ) with the platform. In the +# context of MQTT, APIKEY is linked with the topic used for communication. +APIKEY = 'your_apikey' + +if __name__ == '__main__': + # FIWARE Header + fiware_header = FiwareHeader(service=SERVICE, service_path=SERVICE_PATH) + + # Cleanup at the beginning + clear_all(fiware_header=fiware_header, cb_url=CB_URL, iota_url=IOTA_URL) + + # IoT Agent and OCB Client + iota_client = IoTAClient(url=IOTA_URL, fiware_header=fiware_header) + cb_client = ContextBrokerClient(url=CB_URL, fiware_header=fiware_header) + + # TODO: Setting expression language to JEXL at Service Group level + service_group1 = ServiceGroup(entity_type='Thing', + resource='/iot/json', + apikey=APIKEY, + #... + ) + iota_client.post_group(service_group=service_group1) + + # TODO: Create a device with two attributes 'location' and 'fillingLevel' that use + # expressions. These attributes are based on the attributes 'longitude', + # 'latitude' and 'level', while: + # 1. 'location' is an array with 'longitude' and 'latitude'. + # 2. 'fillingLevel' is 'level' divided by 100 + device1 = Device(device_id="waste_container_001", + entity_name="urn:ngsi-ld:WasteContainer:001", + entity_type="WasteContainer", + transport=TransportProtocol.MQTT, + protocol=PayloadProtocol.IOTA_JSON, + #... + ) + iota_client.post_device(device=device1) + + # TODO: Setting expression language to JEXL at Device level with five attributes, while + # 1. The attribute 'value' (Number) is itself multiplied by 5. The attribute + # 2. 'consumption' (Text) is the trimmed version of the attribute 'spaces' (Text). + # 3. The attribute 'iso_time' (Text) is the current 'timestamp' (Number) transformed into the ISO format. + device2 = Device(device_id="waste_container_002", + entity_name="urn:ngsi-ld:WasteContainer:002", + entity_type="WasteContainer", + transport=TransportProtocol.MQTT, + protocol=PayloadProtocol.IOTA_JSON, + #... + ) + iota_client.post_device(device=device2) + + client = mqtt_client.Client() + client.username_pw_set(username="", password="") + client.connect(MQTT_BROKER_HOST, MQTT_BROKER_PORT) + client.loop_start() + + # TODO: Publish attributes 'level', 'longitude' and 'latitude' of device1 + client.publish(...) + + # TODO: Publish attributes 'value', 'spaces' and 'timestamp' (in ms) of device2 + client.publish(...) + + client.disconnect() + + time.sleep(2) + + # Printing context entities of OCB + for context_entity in cb_client.get_entity_list(entity_types=["WasteContainer"]): + print(context_entity.model_dump_json(indent=4)) + + # Creating two SubWeatherStation entities + entity1 = ContextEntity(id="urn:ngsi-ld:SubWeatherStation:001", + type="SubWeatherStation") + entity1.add_attributes(attrs=[NamedContextAttribute(name="vol", type="Number")]) + cb_client.post_entity(entity1) + + entity2 = ContextEntity(id="urn:ngsi-ld:SubWeatherStation:002", + type="SubWeatherStation") + entity2.add_attributes(attrs=[NamedContextAttribute(name="vol", type="Number")]) + cb_client.post_entity(entity2) + + # TODO: Create a weather station device with multi entity attributes (Number). + # 'v' is multiplied by 100 and is a standard attribute. + # 'v1' and 'v2' are multiplied by 100 and should be linked with entities of + # the SubWeatherStation. + # The name of each attribute is 'vol'. + device3 = Device(device_id="weather_station_001", + entity_name="urn:ngsi-ld:WeatherStation:001", + entity_type="WeatherStation", + transport=TransportProtocol.MQTT, + protocol=PayloadProtocol.IOTA_JSON, + #... + ) + iota_client.post_device(device=device3) + + client = mqtt_client.Client() + client.username_pw_set(username="", password="") + client.connect(MQTT_BROKER_HOST, MQTT_BROKER_PORT) + client.loop_start() + + # TODO: Publish values to all attributes of device3 + client.publish(...) + + client.disconnect() + + time.sleep(2) + + # Printing context entities of OCB + for context_entity in cb_client.get_entity_list(entity_types=["WeatherStation", + "SubWeatherStation"]): + print(context_entity.model_dump_json(indent=4)) diff --git a/tutorials/ngsi_v2/e8_multientity_and_expression_language/e8_multientity_and_expression_language_solution.py b/tutorials/ngsi_v2/e8_multientity_and_expression_language/e8_multientity_and_expression_language_solution.py new file mode 100644 index 00000000..2c20098c --- /dev/null +++ b/tutorials/ngsi_v2/e8_multientity_and_expression_language/e8_multientity_and_expression_language_solution.py @@ -0,0 +1,191 @@ +""" +# # Exercise 8: MultiEntity and Expression Language + +# The MultiEntity plugin allows the devices provisioned in the IoTAgent to map their +attributes to more than one entity, declaring the target entity through the +Configuration or Device provisioning APIs. + +# The IoTAgent Library provides an expression language for measurement transformation, +that can be used to adapt the # information coming from the South Bound APIs to the +information reported to the Context Broker. This is really useful when you need to +adapt measure. + +# There are available two different expression languages jexl and legacy. The +recommended language to use is jexl, which is newer and most powerful. + +# The input sections are marked with 'TODO' + +# #### Steps to complete: +# 1. Setting up the expression language jexl +# 2. Applying the expression language to device attributes +# 3. Testing the expression language via MQTT messages +# 4. Applying the expression language to device attributes in a multi-entity scenario +""" +# Import packages +import time +import datetime + +from filip.clients.ngsi_v2 import IoTAClient, ContextBrokerClient +from filip.models.base import FiwareHeader +from filip.models.ngsi_v2.context import ContextEntity, NamedContextAttribute +from filip.models.ngsi_v2.iot import (Device, ServiceGroup, TransportProtocol, + PayloadProtocol, DeviceAttribute, + ExpressionLanguage) +from filip.utils.cleanup import clear_all +from paho.mqtt import client as mqtt_client + +# Host address of Context Broker +CB_URL = "http://localhost:1026" + +# Host address of IoT-Agent +IOTA_URL = "http://localhost:4041" + +# MQTT Broker +MQTT_BROKER_HOST = "localhost" +MQTT_BROKER_PORT = 1883 + +# FIWARE Service +SERVICE = 'filip_tutorial' +SERVICE_PATH = '/' + +# ToDo: Change the APIKEY to something unique. This represent the "token" +# for IoT devices to connect (send/receive data ) with the platform. In the +# context of MQTT, APIKEY is linked with the topic used for communication. +APIKEY = 'your_apikey' + +if __name__ == '__main__': + # FIWARE Header + fiware_header = FiwareHeader(service=SERVICE, service_path=SERVICE_PATH) + + # Cleanup at the beginning + clear_all(fiware_header=fiware_header, cb_url=CB_URL, iota_url=IOTA_URL) + + # IoT Agent and OCB Client + iota_client = IoTAClient(url=IOTA_URL, fiware_header=fiware_header) + cb_client = ContextBrokerClient(url=CB_URL, fiware_header=fiware_header) + + # TODO: Setting expression language to JEXL at Service Group level + service_group1 = ServiceGroup(entity_type='Thing', + resource='/iot/json', + apikey=APIKEY, + expressionLanguage=ExpressionLanguage.JEXL) + iota_client.post_group(service_group=service_group1) + + # TODO: Create a device with two attributes 'location' and 'fillingLevel' that use + # expressions. These attributes are based on the attributes 'longitude', + # 'latitude' and 'level', while: + # 1. 'location' is an array with 'longitude' and 'latitude'. + # 2. 'fillingLevel' is 'level' divided by 100 + device1 = Device(device_id="waste_container_001", + entity_name="urn:ngsi-ld:WasteContainer:001", + entity_type="WasteContainer", + transport=TransportProtocol.MQTT, + protocol=PayloadProtocol.IOTA_JSON, + attributes=[DeviceAttribute(name="latitude", type="Number"), + DeviceAttribute(name="longitude", type="Number"), + DeviceAttribute(name="level", type="Number"), + DeviceAttribute(name="location", type="Array", + expression="[longitude, latitude]"), + DeviceAttribute(name="fillingLevel", type="Number", + expression="level / 100") + ] + ) + iota_client.post_device(device=device1) + + # TODO: Setting expression language to JEXL at Device level with five attributes, while + # 1. The attribute 'value' (Number) is itself multiplied by 5. The attribute + # 2. 'consumption' (Text) is the trimmed version of the attribute 'spaces' (Text). + # 3. The attribute 'iso_time' (Text) is the current 'timestamp' (Number) transformed into the ISO format. + device2 = Device(device_id="waste_container_002", + entity_name="urn:ngsi-ld:WasteContainer:002", + entity_type="WasteContainer", + transport=TransportProtocol.MQTT, + protocol=PayloadProtocol.IOTA_JSON, + expressionLanguage=ExpressionLanguage.JEXL, + attributes=[DeviceAttribute(name="value", type="Number", + expression="5 * value"), + DeviceAttribute(name="spaces", type="Text"), + DeviceAttribute(name="consumption", type="Text", + expression="spaces|trim"), + DeviceAttribute(name="timestamp", type="Number"), + DeviceAttribute(name="iso_time", type="Text", + expression="timestamp|toisodate"), + ] + ) + iota_client.post_device(device=device2) + + client = mqtt_client.Client() + client.username_pw_set(username="", password="") + client.connect(MQTT_BROKER_HOST, MQTT_BROKER_PORT) + client.loop_start() + + # TODO: Publish attributes 'level', 'longitude' and 'latitude' of device1 + client.publish(topic=f'/json/{APIKEY}/{device1.device_id}/attrs', + payload='{"level": 99, "longitude": 12.0, "latitude": 23.0}') + + # TODO: Publish attributes 'value', 'spaces' and 'timestamp' (in ms) of device2 + client.publish(topic=f'/json/{APIKEY}/{device2.device_id}/attrs', + payload=f'{{ "value": 10, "spaces": " foobar ",' + f' "timestamp": {datetime.datetime.now().timestamp() * 1000} }}') + + client.disconnect() + + time.sleep(2) + + # Printing context entities of OCB + for context_entity in cb_client.get_entity_list(entity_types=["WasteContainer"]): + print(context_entity.model_dump_json(indent=4)) + + # Creating two SubWeatherStation entities + entity1 = ContextEntity(id="urn:ngsi-ld:SubWeatherStation:001", + type="SubWeatherStation") + entity1.add_attributes(attrs=[NamedContextAttribute(name="vol", type="Number")]) + cb_client.post_entity(entity1) + + entity2 = ContextEntity(id="urn:ngsi-ld:SubWeatherStation:002", + type="SubWeatherStation") + entity2.add_attributes(attrs=[NamedContextAttribute(name="vol", type="Number")]) + cb_client.post_entity(entity2) + + # TODO: Create a weather station device with multi entity attributes (Number). + # 'v' is multiplied by 100 and is a standard attribute. + # 'v1' and 'v2' are multiplied by 100 and should be linked with entities of + # the SubWeatherStation. + # The name of each attribute is 'vol'. + device3 = Device(device_id="weather_station_001", + entity_name="urn:ngsi-ld:WeatherStation:001", + entity_type="WeatherStation", + transport=TransportProtocol.MQTT, + protocol=PayloadProtocol.IOTA_JSON, + expressionLanguage=ExpressionLanguage.JEXL, + attributes=[DeviceAttribute(object_id="v1", name="vol", type="Number", + expression="100 * v1", + entity_name="urn:ngsi-ld:SubWeatherStation:001", + entity_type="SubWeatherStation"), + DeviceAttribute(object_id="v2", name="vol", type="Number", + expression="100 * v2", + entity_name="urn:ngsi-ld:SubWeatherStation:002", + entity_type="SubWeatherStation"), + DeviceAttribute(object_id="v", name="vol", type="Number", + expression="100 * v") + ] + ) + iota_client.post_device(device=device3) + + client = mqtt_client.Client() + client.username_pw_set(username="", password="") + client.connect(MQTT_BROKER_HOST, MQTT_BROKER_PORT) + client.loop_start() + + # TODO: Publish values to all attributes of device3 + client.publish(topic=f'/json/{APIKEY}/{device3.device_id}/attrs', + payload='{"v1": 10, "v2": 20, "v": 30}') + + client.disconnect() + + time.sleep(2) + + # Printing context entities of OCB + for context_entity in cb_client.get_entity_list(entity_types=["WeatherStation", + "SubWeatherStation"]): + print(context_entity.model_dump_json(indent=4)) diff --git a/tutorials/ngsi_v2/fiware_env/docker-compose.yml b/tutorials/ngsi_v2/fiware_env/docker-compose.yml new file mode 100644 index 00000000..facb6e4f --- /dev/null +++ b/tutorials/ngsi_v2/fiware_env/docker-compose.yml @@ -0,0 +1,96 @@ + +volumes: + mongodb: ~ + crate-db: ~ + +services: + iot-agent: + image: fiware/iotagent-json:1.26.0 + hostname: iot-agent + container_name: fiware-iot-agent + depends_on: + - mongodb + - mosquitto + expose: + - "4041" + ports: + - "4041:4041" + environment: + - "IOTA_CB_HOST=orion" + - "IOTA_CB_PORT=1026" + - "IOTA_NORTH_PORT=4041" + - "IOTA_REGISTRY_TYPE=mongodb" + - "IOTA_MONGO_HOST=mongodb" + - "IOTA_MONGO_PORT=27017" + - "IOTA_MONGO_DB=iotagent-json" + - "IOTA_PROVIDER_URL=http://iot-agent:4041" + - "IOTA_MQTT_HOST=mosquitto" + - "IOTA_MQTT_PORT=1883" + - "IOTA_DEFAULT_RESOURCE=/iot/json" + - "IOTA_DEFAULT_TRANSPORT=MQTT" + - "IOTA_LOG_LEVEL=DEBUG" + - "IOTA_TIMESTAMP=true" + - "IOTA_CB_NGSI_VERSION=v2" + - "IOTA_AUTOCAST=true" + + mongodb: + image: mongo:4.4 + hostname: mongodb + container_name: db-mongo + ports: + - "27017:27017" + volumes: + - mongodb:/data + + orion: + image: fiware/orion:3.7.0 + hostname: orion + container_name: fiware-orion + depends_on: + - mongodb + expose: + - "1026" + ports: + - "1026:1026" + command: + -dbhost mongodb + -logLevel DEBUG + -logForHumans + + mosquitto: + image: eclipse-mosquitto:2.0.18 + hostname: mosquitto + container_name: mosquitto + expose: + - "1883" + - "9001" + ports: + - "1883:1883" + - "9001:9001" + volumes: + - ./mosquitto.conf:/mosquitto/config/mosquitto.conf + + crate-db: + image: crate:5.4.4 + container_name: cratedb + hostname: crate-db + ports: + - '4200:4200' + - '4300:4300' + command: + crate -Cauth.host_based.enabled=false -Ccluster.name=democluster -Chttp.cors.enabled=true + -Chttp.cors.allow-origin="*" + environment: + - CRATE_HEAP_SIZE=2g + + quantumleap: + image: orchestracities/quantumleap:0.8.3 + container_name: quantumleap + hostname: quantumleap + ports: + - '8668:8668' + depends_on: + - crate-db + environment: + - CRATE_HOST=crate-db + - LOGLEVEL=DEBUG diff --git a/tutorials/ngsi_v2/fiware_env/mosquitto.conf b/tutorials/ngsi_v2/fiware_env/mosquitto.conf new file mode 100644 index 00000000..20351ee7 --- /dev/null +++ b/tutorials/ngsi_v2/fiware_env/mosquitto.conf @@ -0,0 +1,5 @@ + +per_listener_settings false +listener 1883 +log_type all +allow_anonymous true