Skip to content

Files

Latest commit

 

History

History
 
 

legacy_expression_tool

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 

Legacy Expression tool

With release 3.2.0 of IoT Agent Node lib, the legacy expressions language support has been removed. This folder contains the scripts to ease to migrate the legacy expressions.

A manual migration is required, because the expressions need to be converted to JEXL expressions manually.

This python script will help you to migrate the legacy expressions to JEXL expressions. It will search for the legacy expressions in the database and replace them with the new expressions.

This script will search for the legacy expressions in the database and replace them with the new expressions. By default, it runs in dry-run mode, so database is not modified and the output is a series of files that can be used to ease and verify data migration. We you are sure you want to modify the DB, then you have to enable it using the --commit argument.

Installation

This script requires Python 3 to work (it has been tested with Python 3.9 and 3.10).

It is recommended to create virtual env to run this script, installing dependencies on it, this way:

$ virtualenv /path/to/venv
$ source /path/to/venv/bin/active
(venv)$ pin install -r requirements.txt

Output files

The script generates 4 different files each time it is executed:

legacy_expression_ocurrences.json

This file contains information about the document where the legacy expressions are found. It contains the following information for each legacy expression found:

  • _id: The id of the document where the legacy expression is found
  • expression: The legacy expression found
  • type: The type of the property where the legacy expression is found (I.E: active.expression)
  • fiware_service: The fiware service of the document where the legacy expression is found
  • fiware_servicepath: The fiware service path of the document where the legacy expression is found

legacy_expression_list.json

This file contains just a list of all the legacy expressions found in the database. It can be used to build the translation file. The file looks like this:

[
    "${legacy-expression-1}",
    "${legacy-expression-2}",
    ...
    "${legacy-expression-n}"
]

documents_replaced.json

This file contains a list of documents where the legacy expressions are going to be replaced with all the information associated. This file is useful to preview the changes expected to be done in the database before applying them. This file is generated every time the script is executed, so no matter if actual changes are going to be done in the DB (i.e. --commit is used) or not".

documents_backup.json

This file contains a list of documents containing legacy expression that have been backed up. This file is useful to preview the changes expected to be done in the database before modifying it. This file is generated every time the script is executed, so no matter if actual changes are going to be done in the DB (i.e. --commit is used) or not".

Command line arguments

The script can be executed using the following command:

python legacy_expression_tool.py \
    --host <mongodb-host> \
    --port <mongodb-port> \
    --database <mongodb-db> \
    --collection <mongodb-collection> \
    --translation <translation-file>

The list of possible arguments that the scripts accepts are:

Argument Description Default value Mandatory
--mongouri The MongoDB URI to connect to mongodb://localhost:27017/ No
--database The database name to replace the expressions NA Yes
--collection The collection name to replace the expressions NA Yes
--translation The translation dictionary file to replace the expressions translation.json No
--debug Enable debug mode False No
--commit Commit the changes to the database False No
--expressionlanguage What to do with the expression language field. Possibles values: delete, ignore, jexl or jexlall. More detail on this bellow. ignore No
--statistics Print match statistics. Aggregation modes are the possible values: service and subservice service No
--service The fiware service filter to replace the expressions All subservices No
--service-path The fiware service path filter to replace the expressions All subservices No
--deviceid The device id filter to replace the expressions All devices No
--entitytype The entity type filter to replace the expressions All entity types No
--regexservice The fiware service regex filter to replace the expressions All subservices No
--regexservicepath The fiware service path regex filter to replace the expressions All subservices No
--regexdeviceid The device id regex filter to replace the expressions All devices No
--regexentitytype The entity type regex filter to replace the expressions All entity types No

Note that filters (--service, --service-path, --deviceid and --entitytype, and the regex versions) are interpreted in additive way (i.e. like a logical AND).

With regards to --expressionlanguage:

  • delete: changes expressions from legacy to JEXL equivalence in the fields where expressions may be used (based on the translation dictionary specified by --translations). In addition, deletes the expressionLanguage field (no matter its value) in the case it exists in the group/device.
  • ignore: changes expressions from legacy to JEXL equivalence in the fields where expressions may be used (based on the translation dictionary specified by --translations). In addition, it leaves untouched the expressionLanguage field. This may cause inconsistencies, if the value of the expressionLanguage is legacy, as detailed in this section.
  • jexl: changes expressions from legacy to JEXL equivalence in the fields where expressions may be used (based on the translation dictionary specified by --translations). In addition, it set expressionLanguage field to jexl (no matter if the field originally exists in the group/device or not) if some JEXL expression were translated.
  • jexlall: changes expression from legacy to JEXL equivalence in the fields where expressions may be used (based on the translation dictionary specified by --translations). In addition, it set expressionLanguage field to jexl (no matter if the field originally exists in the group/device or not). The difference between jexland jexlall is in the second case, the script is not only looking for documents that contains legacy expressions, it also includes all groups/devices that have expressionLanguage field defined. The expressionLanguage field on those documents (and also on the documents that contains legacy expresions) is set to jexl.

Usage

Getting legacy expressions matches

The recommended way to use the script is to execute it in first stage without passing the commit parameter. This will generate the file legacy_expression_list.json containing all the legacy expressions found in the database. Then, you can use this file to build the translation file.

python legacy_expression_tool.py \
    --host <mongodb-host> \
    --port <mongodb-port> \
    --database <mongodb-db> \
    --collection <mongodb-collection>

This will generate 4 files. At this point, the most relevants are legacy_expression_list.json and legacy_expression_ocurrences.json. The first one contains a list of all the legacy expressions found in the database. The second one contains information about the documents where the legacy expressions are found.

Generating translation file

It is needed to provide a translation file that contains the legacy expressions and the new JEXL expressions to replace them. This is a json file containing an double dimensional array, having the following format:

[
    [
        "legacy-expression-1",
        "legacy-expression-2",
        ...
        "legacy-expression-n"
    ],
    [
        "jexl-expression-1",
        "jexl-expression-2",
        ...
        "jexl-expression-n"
    ]
]

To ease the migration, you can copy the file legacy_expression_list.json generated in previous step and manualy translate the legacy expressions to JEXL expressions. This should be done in the second array of the double dimensional array. The first array should contain the legacy expressions. This mechanism allows to verify that the legacy expressions are replaced correctly.

For more information about JEXL expression language you can check the official documentation and the IoT Agent expression language documentation. You can also check the legacy expression language documentation (deprecated).

Verifying replacements

Once you have the translation file, you can execute the script without passing the commit parameter. This might be helpful to check if the translation file is correct through the output files documents_replaced.json and documents_backup.json. You can run the script using the following command:

python legacy_expression_tool.py \
    --host <mongodb-host> \
    --port <mongodb-port> \
    --database <mongodb-db> \
    --collection <mongodb-collection> \
    --translation <translation-file>

In order to ensure that the replacement would work you have to consider:

  • You didn't get an error while executing. If any expression is not found in the database, the script will raise an error ERROR: Expression not found in translation file: ${@myexpre*100} in document: 123481a8ac03ab212ab9e0c. Note that the script doesn't stop on error (if --commit is enabled, the expressions found in the translation file are translated, remaining in the DB only the ones that result in an error like this, a next pass should be done one the missing expression gets added to the translation document).

  • Comparing files documents_replaced.json and documents_backup.json. The first one contains the documents where the legacy expressions have been replaced with all the information associated. The second one contains a list of documents containing legacy expression that have been backed up. You can compare the files using the following command:

diff documents_replaced.json documents_backup.json

Executing the replacement in the database

If everithing is correct, you can execute the script passing the commit parameter to apply the changes to the database. You only have to provide the --commit parameter to the previous command:

python legacy_expression_tool.py \
    --host <mongodb-host> \
    --port <mongodb-port> \
    --database <mongodb-db> \
    --collection <mongodb-collection> \
    --translation <translation-file> \
    --commit

Output statistics

When the script is executed, it prints some statistics about the matches found. This statistics can be printed filtered by service or subservice. By default, no statistics are printed. You can change it using the --statistics plus the aggregation mode. The possible values are service and subservice.

Found 2 legacy expressions in 2 documents
                                                        _id
service                                         service1 service2 All
expression
${@value*100/total}                                    1        1   2
All                                                    1        1   2

Fields in which expressions may be used

The script implements expression detection and translation in the following fields in group/device documents at DB:

  • active.expression
  • active.entity_name
  • active.reverse
  • attributes.expression
  • attributes.entity_name
  • attributes.expression
  • commands.expression
  • endpoint
  • entityNameExp
  • explicitAttrs

Known issues

Execution with expressionlanguage set to jexlall

When executing the script with expressionlanguage set to jexlall, the script will look for all the documents containing legacy expressions (in some of the expression capable fields) or the existence of the expressionLanguage field. This would change the number of documents found, and the statistics will include extra documents.

Running the script with the option set to jexl would not include such extra documents in statistics.

Replacing expression without setting jexl at group or device level

When executing the script setting --expressionlanguage to ignore (or when --expressionlanguage is not used), the script will not change theexpressionLanguage field in the document. This means that the legacy expressions will be replaced, but the expressionLanguage field will still be set to the default value or legacy. This would make expression evaluation to fail, propagating the value of the attribute as the expression literal to the context broker.

To avoid this, it is recommended use always the --expressionlangauge parameter set to jexl, so doing it a value different from ignore will be used.