Make it flexible to load macros in SSG

[marcusburghardt]

Description:

The CaC/content project uses Jinja2 macros during building time in order to make the content more flexible or easier to be managed in a context with several products. Macros are used in multiple files and should be properly processed in order to return the expected data.

Until recently the SSG library was not expected to be used externally and therefore the path to load macros was hard-coded. This PR brings more flexibility to process yaml files with macros by allowing the macros to be load also from an informed directory.

Rationale:

Allow Jinja2 macros to be consumed when SSG library is called outside the project.

Review Hints:

Here is an example to simulate the issue in a virtual environment:

mkdir ~/tmp
cd ~/tmp
virtualenv ssg_tests
cd ssg_tests
source source bin/activate
pip install git+https://github.com/ComplianceAsCode/content.git

Now you can use the example script from #12797 and you will notice many exceptions similar to what was reported in #12797 (comment).

Once the issue is reproduced, you can try again with the fixes:

pip uninstall ssg
pip install git+https://github.com/marcusburghardt/content.git@ssg_external_macros

No issues are expected now.

Unit tests can be trying with:

python -m pytest tests/unit/ssg-module/test_jinja.py
python -m pytest tests/unit/ssg-module/test_variables.py
python -m pytest tests/unit/ssg-module/test_yaml.py

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[github-actions]

Start a new ephemeral environment with changes proposed in this pull request:

Fedora Environment

Oracle Linux 8 Environment

[Mab879]

While I'm all in favor of adding type hints, the style guide still states all build system code needs to be Python 2 compatible. Last I checked the only hold up was OL7. I have asked in #12046 to get more details on this.

[marcusburghardt]

While I'm all in favor of adding type hints, the style guide still states all build system code needs to be Python 2 compatible. Last I checked the only hold up was OL7. I have asked in #12046 to get more details on this.

Where the type hints were included they are also compatible with Python 2.

[jan-cerny]

But for Python 2 the type hints would have to be in comments, right?
https://peps.python.org/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code

[marcusburghardt]

But for Python 2 the type hints would have to be in comments, right? https://peps.python.org/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code

It is a recommendation, but not mandatory. According to the notes on this link, if moved to type comment, type annotation could not be used together. So would prefer to remove both than keep as type comment considering the intention to drop Python2 support as soon as we can. We also have type hints in other files, so is it worth to remove it on this PR? Let me know what do you think so I can quickly update the PR if necessary. Thanks

[jan-cerny]

But the code won't work with Python 2 if you keep the type hints. The type hint syntax is a new thing in Python 3.5 and wasn't present in Python 2. If you want to have the code compatible with Python 2, you have to put the type hints to comments.

[root@vm-10-0-186-245 ~]# python --version
Python 2.7.5
[root@vm-10-0-186-245 ~]# cat run.py 
#/usr/bin/python

def greeting(name: str) -> str:
    return 'Hello ' + name

greeting("John")
[root@vm-10-0-186-245 ~]# python run.py
  File "run.py", line 3
    def greeting(name: str) -> str:
                     ^
SyntaxError: invalid syntax

[marcusburghardt]

But the code won't work with Python 2 if you keep the type hints. The type hint syntax is a new thing in Python 3.5 and wasn't present in Python 2. If you want to have the code compatible with Python 2, you have to put the type hints to comments.

[root@vm-10-0-186-245 ~]# python --version
Python 2.7.5
[root@vm-10-0-186-245 ~]# cat run.py 
#/usr/bin/python

def greeting(name: str) -> str:
    return 'Hello ' + name

greeting("John")
[root@vm-10-0-186-245 ~]# python run.py
  File "run.py", line 3
    def greeting(name: str) -> str:
                     ^
SyntaxError: invalid syntax

Ok. Let me quickly remove in this specific PR so we can move forward. I will comment in the discussion to confirm if we are already good to officially remove Python 2 support. The informed date there was 2014/12.

[jan-cerny]

yeah, it would be great to drop the python2 support soon, the type hints feature is a nice reason

[marcusburghardt]

yeah, it would be great to drop the python2 support soon, the type hints feature is a nice reason

Type hints removed from functions changed in this PR @jan-cerny .

[codeclimate]

Code Climate has analyzed commit 371915d and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (50% is the threshold).

This pull request will bring the total coverage in the repository to 61.9% (0.1% change).

View more on Code Climate.