Hot Fix for Pydantic (#46) #379
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Migrated the runtime to pydantic 2.0 * Generation and build tools now refer to teh new version of the rosetta runtime. * require pydantic version to be >= 2 * Function Generator changes * Fix missing files * Use the fully qualified name for datetime objects as to avoid name clash with the attribute name. * pydantic v2 support - reverted the model to use the v1 from pydantic v2 as recursive definitions still fail in v2 despite being supported. * Fixed failing tests after change to use fqn for datetime types * rebuilt the runtime whl package with the latest changes * fix defective code generation * change pydantic and runtime required versions * fix issue with generating imports for base types. * removed the docstring from the functions as xtend doesn't indent it correctly. * switched off the function generation * Added Python unit testing * added an example function (`AddBusinessDays`) - the code generation of the expressions can be used as is and only a small change to `_resolve_rosetta_attr` would be required to make the so generated code to work. * Temporal func implementation with some errors * Function Generation and Expression Operators * Added the ability to register condition functions in a local registry (for use in the functions generation) Added the ability to decorate functions as replaceable and provide runtime replacements * Merge develop work * Migrated to Pydantic 2.x.x. * fixed the dict comparison function and used model_dum_json as pydantic 2.x.x compliant * updated test and python build including adding corrected EUR-Vanilla...json * 2nd commit of changed testing code * converted indentation to spaces to support Python * first pass cleanup of function generation. used of spaces instead of tabs * Fix the serialization of enums * removed unnecessary toFirstUpper call in toPythonBasicType * Small refactoring of PythonFunctionGenerator - Moved duplicative toPythonBasicType to PythonTranslator. - removed unused imports - simplified logic * Cr fixes (#8) * some conditions related fixes * fix the recursive condition execution * - more sane logging in validate_conditions - replace deprecated call - let the logging be displayed in the validation test --------- Co-authored-by: Plamen Neykov <[email protected]> * fix various inconsistencies how None/[] values are treated (from Rosetta point if view, they are the same) (#9) * updated the runtime whl to reflect the latest changes * _resolve_rosetta_attr resolves correctly when the object has meta data. * ongoing code cleanup * clean up including removal of tabs and consistent formatting * Revert "clean up including removal of tabs and consistent formatting" This reverts commit 6dcc36a. * code cleanup / formatting and removal of functions * implementation unit test * delete file 2.0.0 * code refactoring * park changed code * Revert "code refactoring" This reverts commit ae0610a. * reset to working code from develop * added import sys to createImportsFunc * refactored generating python type from Rosetta type * clean up to remove warnings * further refactoring - consolidated transform of type in PythonModelObjectGenerator - in progress setup for independent python generation in PythonFilesGeneratorTest * More refactoring of basic types, cleanup and removal of hard specification of "import cdm" * removed commenting code and added duplicate rosetta files to unit test source * Majority of expression and operation tests implemented * small fix incorrect merging * Fix Function Conditions * Fix Function JUnit tests * Fix PyTest * Use pydantic V2 configuration style. Attributes containing models will be revalidated when model_validate is invoked. * Updated runtime whl. * read_trade fix * revised run_tests.sh and replaced tabs in pom * fixed pom error * removed unused imports * Set the rosetta runtime version to 2.0.0. Generate py.typed in the namespace root for mypy. * use runtime 2.0.0 * align to the changed rosetta_runtime numbering * Clean up replacing tabs with spaces * Fix Junit Test * clean up python unit tests to conform to Pydantic 2.x and dockerfile * Clean-up * Add rosetta files testing * removed cdm rosetta from python_unit_tests * additional clean up - pom * clean up run_tests * Delete commented code * Pre-PR clean up including the build and testing shell scripts * More pre-release cleanup including release notes * clean up of build and test in prep for PR * test dir structure organized for multiple invocations * updated test config to include cdm repo info * release notes cleanup * pre-release testing update * adding source rosetta back for testing (by change to .gitignore) * Fix Boolean issue * Updated the json test file to conform to the CDM v5.8.0 * small refactoring of test_validation.py * clean up validation test * Update RELEASE.md * prep for release - removed rosetta files and disabled file generator * Merge V2 changes into master (#20) * Migrated the runtime to pydantic 2.0 * Generation and build tools now refer to teh new version of the rosetta runtime. * require pydantic version to be >= 2 * Function Generator changes * Fix missing files * Use the fully qualified name for datetime objects as to avoid name clash with the attribute name. * pydantic v2 support - reverted the model to use the v1 from pydantic v2 as recursive definitions still fail in v2 despite being supported. * Fixed failing tests after change to use fqn for datetime types * rebuilt the runtime whl package with the latest changes * fix defective code generation * change pydantic and runtime required versions * fix issue with generating imports for base types. * removed the docstring from the functions as xtend doesn't indent it correctly. * switched off the function generation * Added Python unit testing * added an example function (`AddBusinessDays`) - the code generation of the expressions can be used as is and only a small change to `_resolve_rosetta_attr` would be required to make the so generated code to work. * Temporal func implementation with some errors * Function Generation and Expression Operators * Added the ability to register condition functions in a local registry (for use in the functions generation) Added the ability to decorate functions as replaceable and provide runtime replacements * Merge develop work * Migrated to Pydantic 2.x.x. * fixed the dict comparison function and used model_dum_json as pydantic 2.x.x compliant * updated test and python build including adding corrected EUR-Vanilla...json * 2nd commit of changed testing code * converted indentation to spaces to support Python * first pass cleanup of function generation. used of spaces instead of tabs * Fix the serialization of enums * removed unnecessary toFirstUpper call in toPythonBasicType * Small refactoring of PythonFunctionGenerator - Moved duplicative toPythonBasicType to PythonTranslator. - removed unused imports - simplified logic * Cr fixes (#8) * some conditions related fixes * fix the recursive condition execution * - more sane logging in validate_conditions - replace deprecated call - let the logging be displayed in the validation test --------- Co-authored-by: Plamen Neykov <[email protected]> * fix various inconsistencies how None/[] values are treated (from Rosetta point if view, they are the same) (#9) * updated the runtime whl to reflect the latest changes * _resolve_rosetta_attr resolves correctly when the object has meta data. * ongoing code cleanup * clean up including removal of tabs and consistent formatting * Revert "clean up including removal of tabs and consistent formatting" This reverts commit 6dcc36a. * code cleanup / formatting and removal of functions * implementation unit test * delete file 2.0.0 * code refactoring * park changed code * Revert "code refactoring" This reverts commit ae0610a. * reset to working code from develop * added import sys to createImportsFunc * refactored generating python type from Rosetta type * clean up to remove warnings * further refactoring - consolidated transform of type in PythonModelObjectGenerator - in progress setup for independent python generation in PythonFilesGeneratorTest * More refactoring of basic types, cleanup and removal of hard specification of "import cdm" * removed commenting code and added duplicate rosetta files to unit test source * Majority of expression and operation tests implemented * small fix incorrect merging * Fix Function Conditions * Fix Function JUnit tests * Fix PyTest * Use pydantic V2 configuration style. Attributes containing models will be revalidated when model_validate is invoked. * Updated runtime whl. * read_trade fix * revised run_tests.sh and replaced tabs in pom * fixed pom error * removed unused imports * Set the rosetta runtime version to 2.0.0. Generate py.typed in the namespace root for mypy. * use runtime 2.0.0 * align to the changed rosetta_runtime numbering * Clean up replacing tabs with spaces * Fix Junit Test * clean up python unit tests to conform to Pydantic 2.x and dockerfile * Clean-up * Add rosetta files testing * removed cdm rosetta from python_unit_tests * additional clean up - pom * clean up run_tests * Delete commented code * Pre-PR clean up including the build and testing shell scripts * More pre-release cleanup including release notes * clean up of build and test in prep for PR * test dir structure organized for multiple invocations * updated test config to include cdm repo info * release notes cleanup * pre-release testing update * adding source rosetta back for testing (by change to .gitignore) * Fix Boolean issue * Updated the json test file to conform to the CDM v5.8.0 * small refactoring of test_validation.py * clean up validation test * Update RELEASE.md * prep for release - removed rosetta files and disabled file generator --------- Co-authored-by: Plamen Neykov <[email protected]> Co-authored-by: David Carrascosa <[email protected]> Co-authored-by: dan <[email protected]> Co-authored-by: dcarrascosa00 <[email protected]> Co-authored-by: Plamen Neykov <[email protected]> Co-authored-by: Plamen Neykov <[email protected]> Co-authored-by: priyanka-amarnani <[email protected]> * Delete python/prototypes/example_function directory (#21) * Fix import issue * Fix format file * fix for _global and update bond-option unit test source to 5.10.0 version * Account in rosetta_resolve_attr for the name mangling of reserved python keywords. Add also a rosetta test for this case. * refactored file generation * updates to ensure that __init__.py is added for all generated python * Rosetta driven unit tests packaged and installed by run_tests.sh * Resolve key word defect * merge clean up * grouped tests into cdm and rosetta * pre-release cleanup * refactored into separate rosetta and cdm tests * revised name mangling to check for python keywords, soft keywords or attrib beginning with "_" * refactor mangling use of keywords, soft keywords or begins with "_". also dummy placeholders for missing operators * Implemented ToStringOperation and ToEnumOperation * Fixed an issue when stringifying an Enum with non string values. Fixed a missing import and added a test. * added type annotations and moved the standard import keyword to the front * moved the stand-alone runtime tests in their own directory * Python Unit tests added to codefresh process * refactoring of unit tests * Fixed the version tuple in version.py * Simplified the condition wrappers (also speeds up execution). * moved the dsl tests in the package rosetta_dsl to avoid some namespace clashes with rosetta.runtime module * fixed an undecidability issue with AttributeWithAddress * Merge back master into develop (#31) * Upgrade of Python Generator (REGnosys#290) # _Python Generator v2_ _What is being released?_ This release contains a number of upgrades to the Python generator: - Migration to Pydantic 2.x - More comprehensive support for Rosetta's operators (see full list below) - Resolves the defect exposed by [PR 2766](finos/common-domain-model#2766) - Includes an update to the Python Rosetta runtime library used to encapsulate the Pydantic support (now version 2.0.0) Future releases will: - Enable generation of Python for Rosetta-defined functions - Provide more complete support for re-use of data designated as "meta" ## New Operators ### Basic Expressions | Rosetta Syntax | Description | Supported | |:---|:---|:---:| | RosettaExpression | The base type for all expressions in the Rosetta DSL.| | RosettaLiteral | Represents constant values directly in the code. | | RosettaBooleanLiteral | Represents a true or false value. | :white_check_mark: | | RosettaIntLiteral | Represents an integer constant. | :white_check_mark:| | RosettaNumberLiteral | Represents a numeric constant whether an integer or a decimal. | :white_check_mark:| | RosettaStringLiteral | Represents a sequence of characters, enclosed in quotes. | :white_check_mark:| | ListLiteral | Represents a fixed list of elements, e.g., \[1, 2, 3]. | :white_check_mark: | ### Reference and Feature Call Expressions | Rosetta Syntax | Description | Supported | |:---|:---|:---:| | RosettaFeatureCall | Calls a feature (attribute or method) of an object or data type. | :white_check_mark: | | RosettaReference | A reference to another part of the model or DSL. | :white_check_mark: | | RosettaSymbolReference | A reference to a symbolic constant or a variable. | :white_check_mark: | | RosettaEnumValue | A specific value from an enumeration. | :white_check_mark:| | RosettaEnumValueReference | A reference to a specific value within an enumeration. | :white_check_mark:| | RosettaImplicitVariable | An implicitly declared variable that is available in a certain context. | :white_check_mark:| ### Operation Expressions | Rosetta Syntax | Description | Supported| |:---|:---|:---:| | RosettaUnaryOperation | An operation that takes a single operand, like negation. | :x: | | RosettaBinaryOperation | An operation that takes two operands, such as addition or logical 'and'. | :white_check_mark:| | RosettaCountOperation | Counts the number of elements in a collection. | :white_check_mark:| | ArithmeticOperation | A mathematical calculation like addition, subtraction, multiplication, or division.| | ComparisonOperation | Compares two expressions, such as greater-than or less-than. | :x: | | EqualityOperation | Checks for equality (or inequality) between two expressions. | :x: | | LogicalOperation | Logical operations like AND, OR, and NOT. | :white_check_mark: | | ChoiceOperation | Allows for selection between multiple potential values or paths. | :white_check_mark:| | FilterOperation | Filters a collection based on a predicate, returning only those elements that satisfy the predicate. | :white_check_mark: | | MapOperation | Transforms each element of a collection according to a specified function. | :white_check_mark: | | ReduceOperation | Combines the elements of a collection down to a single value. | :x: | | SumOperation | Adds together all elements in a collection. | :white_check_mark: | | MaxOperation | Determines the maximum value within a collection. | :white_check_mark: | | MinOperation | Determines the minimum value within a collection. | :white_check_mark: | | DistinctOperation | Filters a collection to only include distinct elements. | :white_check_mark:| | SortOperation | Sorts the elements of a collection. | :white_check_mark: | | FirstOperation | Retrieves the first element from a collection. | :white_check_mark: | | LastOperation | Retrieves the last element from a collection. | :white_check_mark: | | FlattenOperation | Flattens nested collections into a single collection. | :white_check_mark:| | ReverseOperation | Reverses the order of elements in a collection. | :white_check_mark: | | JoinOperation | Concatenates a collection of strings into a single string, with an optional delimiter. | :white_check_mark: | | AsKeyOperation | Designates a value as a key in a key-value pair. | :white_check_mark: | | ThenOperation | Chains multiple operations, where the output of one is the input to the next. | :white_check_mark: | ### Conditional and Specialized Expressions | Rosetta Syntax | Description | Supported | |:---|:---|:---:| | RosettaConditionalExpression | An expression that evaluates a condition and branches accordingly. | :white_check_mark: | | RosettaContainsExpression | Checks if a collection contains a given element. | :white_check_mark: | | RosettaDisjointExpression | Determines if two collections have no elements in common. | :white_check_mark: | | RosettaExistsExpression | Checks if a given value or property exists. | :white_check_mark: | | RosettaAbsentExpression | Represents the absence of a value. | :white_check_mark: | | RosettaOnlyElement | Ensures a collection contains exactly one element and retrieves it. | :white_check_mark: | | RosettaOnlyExistsExpression | Asserts that only one of the given conditions or expressions is true. | :white_check_mark: | | OneOfOperation | Ensures only one of several possible values is chosen. | :white_check_mark: | | RosettaConstructorExpression | Creates new objects in Rosetta, translating to Python constructors or dictionaries | :white_check_mark: | ### Conversion Operations | Rosetta Syntax | Description | Supported | |:---|:---|:---:| | ToEnumOperation | Converts a value to an enumeration type. | :x: | | ToIntOperation | Converts a value to an integer. | :white_check_mark: | | ToNumberOperation | Converts a value to a number (e.g., BigDecimal in Java). | :white_check_mark: | | ToStringOperation | Converts a value to a string. | :white_check_mark: | | ToTimeOperation | Converts a value to a time representation. Python Generator expressions used | :x: | * Updated DSL version (REGnosys#291) * Updated DSL version * Updated bundle version * task-4383: Update codefresh.yml * task-4383: Migrate to Maven Central * task-4383: revert version update * task-4383: update codefresh.yml * task-4383: update codefresh.yml * task-4383: fix substitution in codefresh.yml * task-4383: more updates to codefresh.yml * task-4383: fix path in settings.xml * Story-845: Migrate to Maven Central - Releases (REGnosys#293) * task-4563: fix key leak * Update DSL version to 9.8.5 (REGnosys#299) * update dsl version (REGnosys#300) * Updated DSL (REGnosys#303) * Support for mangled names and additional operators (REGnosys#307) Co-authored-by: dschwartznyc <[email protected]> --------- Co-authored-by: dschwartznyc <[email protected]> Co-authored-by: SimonCockx <[email protected]> Co-authored-by: Jayasri Radhakrishnan <[email protected]> Co-authored-by: JayasriR <[email protected]> Co-authored-by: James Annesley <[email protected]> Co-authored-by: James Annesley <[email protected]> Co-authored-by: Hugo Hills <[email protected]> Co-authored-by: David Al-Kanani <[email protected]> * interim commit - need to resolve paths to attributes * Delete python/run.txt * interim support for choice aliases (syntax to be revised) * fixed failing unit tests (bc of formatting) * Added the required by the deep path operator `rosetta_resolve_deep_attr` function. * Added `rosetta_resolve_deep_attr` in `__all__` * skip run.txt * update to create choice aliases inline with target (includes debugging code) * updated to invoke the deep path operator * added deep path python unit tests * interim update in prep for removal of deprecations * for testing * removed tabs * removed debugging code * test for failure on use of the switch operator * Pydantic hot fix (#36) * pydantic dependency set to >=2.6,<2.10 and test script cleanup * clean up tests * another clean up of deprecated references * gitignore clean up * more cleanup * more more cleanup * more more more cleanup * more ... cleanup * more deprecated code cleanup * cleanup continues * updated to remove deprecations * fix pydantic toml versions and clean up build script naming * restore run_test.sh cleanup of env when finished * update checkstyle * clean up run_tests.sh * test clean up * added build and test instructions * fixed the metadata treatment and relaxed the requirements for a replacement function - now it has to have the same number of parameters and the names must match, but the types of the parameters are excluded from the check (for now). * added a fix for a bad model in the json when running `test_read_and_validate` * clean up tests and parent POM * pre-release changes * more pre-release clean up * update test scripts to only run rosetta tests (remove cdm tests) * added cheat sheet and updated cdm unit test * updated with support and unit tests for min and max * updated Function dependency checking * Fixed defects in the generator and runtime functions * Fixed defects in the generator and runtime functions * - unified both check_one_of functions in one in the runtime and amended the generation and tests. - removed duplicated functions from the runtime - fixed inconsistent use of tabs in the runtime python source - removed and unused parameter form rosetta_filter * integrate new function dependency checking * removed the obsolete check_one_of_constraint method from the BaseDataClass * sample json clean up * Added lambda boolean to min/max * uodate rosetta_runtime * pre refactoring of function generator * refactored FunctionDependencyProvider * clean up cdm serialization unit test * Switch operator implementation and unit test * Updated the string compare test and deleted an obsolete function * Updated the string compare test and deleted an obsolete function * release prep * runtime cleanup * release clean up * udpate to README * remove checkstyle * interim clean up * celan POM * java code clean up * gitgnore cleanup * remove RELEASE.md * clean up build_cdm.sh * additional clean up * first update to current dsl version * clean up to support current DSL * REGNosys changes * readme update * uitl class clean up * convert Util from xtend to java * read me clean up * minor clean up * remove serialization pre-work * dsl and bundle update --------- Co-authored-by: Plamen Neykov <[email protected]> Co-authored-by: David Carrascosa <[email protected]> Co-authored-by: dan <[email protected]> Co-authored-by: dcarrascosa00 <[email protected]> Co-authored-by: Plamen Neykov <[email protected]> Co-authored-by: Plamen Neykov <[email protected]> Co-authored-by: priyanka-amarnani <[email protected]> Co-authored-by: SimonCockx <[email protected]> Co-authored-by: Jayasri Radhakrishnan <[email protected]> Co-authored-by: JayasriR <[email protected]> Co-authored-by: James Annesley <[email protected]> Co-authored-by: James Annesley <[email protected]> Co-authored-by: Hugo Hills <[email protected]> Co-authored-by: David Al-Kanani <[email protected]>
hugohills-regnosys
approved these changes
Jan 15, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is being released?
This release contains hot fixes to make the Python generator compatible with CDM 5.x.x. The changes will
not address CDM 6 compatibility.
#304
#302
#366
Partially Closed: #246