Long-term draft topic: Vsc lib with fixes #31

gunnarx · 2022-08-15T15:12:53Z

Fixed according to feedback on PR #30. I didn't force push that branch without @magnusfeuer's permission. In any case, I recommend we also squash these small fixes into the original commit, and that we should merge the PR (it still goes to separate branch). What do you think?

erikbosch · 2022-08-15T17:08:09Z

vsclib/LICENSE

@@ -0,0 +1,373 @@
+Mozilla Public License Version 2.0


This is the same license as in root repo, right? So why do we need this file.

erikbosch · 2022-09-16T07:05:41Z

vsclib/setup.py

+    description="VSC Generator Library",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    url="https://github.com/COVESA/vsclib",


Will this URL ever exist? Or should it be vsc-tools?

Yes, as of now it seems like it should be vsc-tools.

erikbosch · 2022-09-16T07:10:09Z

vsclib/setup.py

+    classifiers=[
+        "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: 3.8",
+        "License :: Other/Proprietary License",


As we use MPL2 - is the license statement here correct? Compare with e.g. https://github.com/mozilla/libmozdata/blob/master/setup.py

Also, here we explictly state Python 3.7 and 3.8, but in buildcheck we use Python 3.10. Is it needed to specify Python version here. (In the long run, but not now, we should preferably perform tests with all Python versions we claim to support. I.e. if we say here that Python 3.7 is sufficient then we shall have a build that verifies that 3.7 actually is sufficient)

I agree, the license info looks wrong to me.

Yeah the python version in the buildcheck has evolved over time. I seem to recall that it started with a language feature we used that required 3.9, and then the Ubuntu version of the GitHub CI environment underwent some change so it did not support the version we demanded, so it was then changed to 3.10 for the CI to run OK.
In any case, good catch - we should make this consistent and also avoid overspecifying such details.

* Add python setuptools setup.py script. * Move python code to the vsc folder. * Move templates folder to the vsc folder. * Install vscgen tool to simplify codegenerator use via cmd. * Update github actions (use python 3.10 and vscgen). Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

* Implement vehicle service catalog AST using python dataclasses. * Implement parsing of YAML vsc format into vsc AST format. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

* Add simple tests for newly written dataclasses AST. * Add new example YAML files input2.yaml and input3.yaml. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

Take over typic gitignore for python projects from https://github.com/github/gitignore/blob/main/Python.gitignore Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

All the previous code in this file, to read YAML into an AST tree, is now superseded by vsc_ast.py. There are a few files and scripts referring to this file so at the moment it remains as a wrapper around vst_ast but soon it should probably be replaced completely.

(should be no functional change)

- Methods need to be inside a namespace - includes: should be a list - Change one test to output name of main file instead of first namespace See also previous comment

From what I can tell, the new parsing function instantiates all needed fields of all objects in the AST tree (or leaves the field as the default (= None), if something is Optional and not provided by the input YAML. I therefore removed all the post_init functions, here as a separate commit in case we need to test further, rollback, or something. unit-tests and basic template generation seem to run fine (except for those templates that need update, but that is known)

In vscgen tool template argument becomes optional. When vscgen <yaml.file> called it would use default templates from vsc.templates.default_templates dictionary. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

After short discussion we have decided to bring back the AST node. This node is always a root, but inherits everything from the Namespace class. And vsc_parser.gen() function now accepts object of any type. Later we might want to introduce some checks for the passed node, like assert dataclass.is_dataclass(node). Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

Moving all code related to YAML processing to the vsc_parser.py from vsc_ast.py to keep in vsc_ast module only AST nodes. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

Use Any instead of Generic[T] for gen() function. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

Improve setup.py script to properly package and install files inside the templates folder. Now setup.py will install the *.tpl and *.html files into the respective python3 packages directory. This allows to use module independently. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

An attempt to reverse dependencies by moving all the jinja2 related code to the vsc.templates module. This makes vsc_generator.py more abstract and in theory removes dependency on the concrete template engine. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

Making description field empty by default instead of None. Removing Optional[..] from the Lists and using default_factory assigning empty lists by default to avoid additional checks in the templates code. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

Bring back Optional[List[...]] and Optional[str] to some of the fields to bring AST closer to the specification. It would not change the runtime behaviour, lists would have default value [], etc. Fix arraysize type in some of the classes from Optional[str] to Optional[int]. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

Change python version from 3.10.6 to 3.10.7. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

Comments added to the AST classes, using information from the VSC specification and format of python docstrings. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

I noticed some discrepancy in the readme file and rawly fixed all the incosistencies I have been able to find out. Maybe there are bigger changes needed, but I will leave it for the future PRs. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

Signed-off-by: Erik Jaegervall <[email protected]>

gunnarx · 2022-10-31T16:57:50Z

Corrected the setup.py file according to the remarks noted above.

This configures `dacite` to throw an exception if the input (yaml-dict) contains any unexpected data items (usually coming from a misspelled or misplaced keyword in the YAML input). Then we catch the exception and output a message that is as useful as we can, given the information. Signed-off-by: Gunnar Andersson <gunnar_dev@[email protected]>

- Each test is now stored in a subdir under tests. - Each test subdir must be named test.<something>. - gen_test.py loops over those directories and performs the same test as before (generate the given 'input.py' using 'template' and compare to 'result') Signed-off-by: Gunnar Andersson <gunnar@[email protected]>

The test only checks that the expected exception is raised on illegal input. Signed-off-by: Gunnar Andersson <gunnar@[email protected]>

It seems that GitHub bumps the patch level number frequently. (right now from 3.10.7 to 3.10.8) Let's go with 3.10 instead of the more specific 3.10.7 to see if this works more long term.

… a VSC tree Signed-off-by: Magnus Feuer <[email protected]>

Simple fixes according to PR feedback - can be squashed during merge.

byteBuffer is the name used in the current specification

(minor, can be squashed during merge)

It it self-evident from the LICENSE file that this was intended to be MPLv2 licensed and the setup.py script should reflect this. (When it becomes part of vsc-tools, then this setup.py might even be superfluous and therefore removed)

gunnarx · 2022-11-14T10:04:09Z

Just rebased on master as a starting point, because there were 34 new commits.

gunnar-mb · 2024-01-23T17:10:27Z

Reference: #100 implements a more rudimentary version of this support for creating IFEX (previously VSC) definition from within a python program. The constructors provided by #100 are derived directly from the current IFEX language model.

The original vsc_lib does a bit more by including code that does type resolution, checking if types are visible through the namespace visibility rules, etc.

However, at least the construction of an IFEX interface tree representation is likely covered adequately by #100 now, with the advantage of it being up to date (and automatically up to date when/if the model changes).

gunnarx force-pushed the vsc_lib_with_fixes branch 2 times, most recently from ab2150d to a559637 Compare August 15, 2022 15:20

erikbosch reviewed Aug 15, 2022

View reviewed changes

This was referenced Aug 19, 2022

Implement VSC AST using python dataclasses #32

Merged

README: Update all setup/installation instructions #33

Merged

gunnarx mentioned this pull request Sep 15, 2022

Initial code check in of vsclib #30

Closed

erikbosch reviewed Sep 16, 2022

View reviewed changes

miketsukerman and others added 22 commits September 21, 2022 15:56

vsc_ast: use python dataclasses for AST

deb95e9

* Implement vehicle service catalog AST using python dataclasses. * Implement parsing of YAML vsc format into vsc AST format. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

get_tests: update vsc-tools python tests

a918d9d

* Add simple tests for newly written dataclasses AST. * Add new example YAML files input2.yaml and input3.yaml. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

.gitignore: extend list by python default temporary files

cb58613

Take over typic gitignore for python projects from https://github.com/github/gitignore/blob/main/Python.gitignore Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

vsc_ast: Rewrite main parsing function

6fcff3e

vsc_ast: Complete the AST classes

b47b9eb

vsc_parser: Delegate to new vsc_ast module

0eadd84

vsc_generator: Update to match new vst_ast parser

0ac8bef

vsc_generator: Formatting and cleaning

a000667

(should be no functional change)

tests: Update for current language model

916627b

- Methods need to be inside a namespace - includes: should be a list - Change one test to output name of main file instead of first namespace See also previous comment

templates: Update to match current language model

b329dd5

GitHub Actions Temporary branch change

a49ec43

vsc_ast: move parsing code away from vsc_ast to vsc_parser

4f53efa

Moving all code related to YAML processing to the vsc_parser.py from vsc_ast.py to keep in vsc_ast module only AST nodes. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

vsc_generator: make algorithms more abstract

f5cfeb2

Use Any instead of Generic[T] for gen() function. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

miketsukerman added 4 commits September 21, 2022 15:56

.github: update github actions configuration

47cc325

Change python version from 3.10.6 to 3.10.7. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

vcs_ast: add comments to the AST classes

b176b03

Comments added to the AST classes, using information from the VSC specification and format of python docstrings. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

copyright: fix license headers

38d28a9

Remove line "#All right reserved" from the files added by previous MR. Signed-off-by: Mikhail Tsukerman <[email protected]> Signed-off-by: Magnus Feuer <[email protected]>

gunnarx self-assigned this Sep 25, 2022

gunnarx marked this pull request as draft September 26, 2022 14:59

erikbosch and others added 3 commits September 26, 2022 19:55

Fix documentation regression

7a70f4c

Signed-off-by: Erik Jaegervall <[email protected]>

README: Update all setup/installation instructions

6e1b25b

Fix template regressions

7e30d68

Signed-off-by: Erik Jaegervall <[email protected]>

gunnarx force-pushed the vsc_lib_with_fixes branch from a5a68e9 to 7fcc47f Compare October 31, 2022 16:52

Gunnar Andersson and others added 5 commits October 31, 2022 19:22

Add test for illegal keys

df2051d

The test only checks that the expected exception is raised on illegal input. Signed-off-by: Gunnar Andersson <gunnar@[email protected]>

GitHub: Change python 3.10.8 -

44f38a1

It seems that GitHub bumps the patch level number frequently. (right now from 3.10.7 to 3.10.8) Let's go with 3.10 instead of the more specific 3.10.7 to see if this works more long term.

Add a README inside tests for developer information

011c1a8

gunnarx mentioned this pull request Nov 7, 2022

Combine vsc-lib and ifex tools (ifex_ast) #45

Closed

magnusfeuer-mbition and others added 7 commits November 14, 2022 11:02

Initial checkin of vsclib code to programatically build up and verify…

f4dbe69

… a VSC tree Signed-off-by: Magnus Feuer <[email protected]>

vsc_lib: spelling fixes

1d96955

Simple fixes according to PR feedback - can be squashed during merge.

vsc_lib binary->byteBuffer

97dba61

byteBuffer is the name used in the current specification

vsc_lib: Change exit code

4bc0e10

(minor, can be squashed during merge)

vsclib/setup.py: Correct URL

26d98f7

vsclib/setup.py: Use python 3.10

5b4d49f

vsclib/setup.py: Change license information.

8ef8282

It it self-evident from the LICENSE file that this was intended to be MPLv2 licensed and the setup.py script should reflect this. (When it becomes part of vsc-tools, then this setup.py might even be superfluous and therefore removed)

gunnarx force-pushed the vsc_lib_with_fixes branch from 19dadc7 to 8ef8282 Compare November 14, 2022 10:03

gunnar-mb changed the title ~~Vsc lib with fixes~~ Long-term draft topic: Vsc lib with fixes Jan 23, 2024

gunnar-mb deleted the branch COVESA:vsclib June 4, 2024 15:43

gunnar-mb closed this Jun 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Long-term draft topic: Vsc lib with fixes #31

Long-term draft topic: Vsc lib with fixes #31

gunnarx commented Aug 15, 2022

erikbosch Aug 15, 2022

erikbosch Sep 16, 2022

gunnarx Sep 25, 2022

erikbosch Sep 16, 2022

gunnarx Sep 25, 2022 •

edited

Loading

gunnarx commented Oct 31, 2022

gunnarx commented Nov 14, 2022

gunnar-mb commented Jan 23, 2024 •

edited

Loading

Long-term draft topic: Vsc lib with fixes #31

Long-term draft topic: Vsc lib with fixes #31

Conversation

gunnarx commented Aug 15, 2022

erikbosch Aug 15, 2022

Choose a reason for hiding this comment

erikbosch Sep 16, 2022

Choose a reason for hiding this comment

gunnarx Sep 25, 2022

Choose a reason for hiding this comment

erikbosch Sep 16, 2022

Choose a reason for hiding this comment

gunnarx Sep 25, 2022 • edited Loading

Choose a reason for hiding this comment

gunnarx commented Oct 31, 2022

gunnarx commented Nov 14, 2022

gunnar-mb commented Jan 23, 2024 • edited Loading

gunnarx Sep 25, 2022 •

edited

Loading

gunnar-mb commented Jan 23, 2024 •

edited

Loading