feat: Validate dataset properties in jobs #1473
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Documentation is now added to SciCatProject/documentation#56 |
|
The unit tests pass because I'm manually mocking the datasetService, so the core functionality works. The mocha tests fail because the datasetService doesn't get injected properly by nest. I need to figure out how to properly initialize the |
sbliven
force-pushed
the
validateaction
branch
from
c689643 to
be8fe75
Compare
During a `create` operation, the `validate` action can take a `datasets` option listing path/schema pairs that will be applied to any datasets in jobParams.datasetList. Datasets are fetched from the database during the DTO validation step if needed. Validation of archive/retrieve/public lifecycle properties is no longer done automatically, but must be configured in the jobConfig file. Details: - Separate ValidateCreateJobAction (create) and ValidateJobAction (update) - Fix 'Internal Server Error' from job controller if a session timed out. If the session times out then only public jobs will be visible. - Update documentation for validate action - Remove checkDatasetState from the job controller. This must now be configured with a validate action - Add unit test - Remove hard-coded job types and dataset states
sbliven
force-pushed
the
validateaction
branch
from
be8fe75 to
5e32d75
Compare
|
I rebased this onto the current release-jobs. Tests are currently failing due to a circular dependency in the module DI. I'm looking into how to fix it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Allows jobs to validate properties of the dataset.
Motivation
The
validateaction currently allows operators to enforce attributes of the DTO.However, many jobs depend on
jobParams.datasetListsto link a job to certain datasets. This allows validation of properties on datasets associated with the dataset.The motivating example would be checking datasetLifecycle properties for certain jobs. These are currently hard-coded for special job types. This would allow them to be applied to custom job types as well.
Changes:
datasetsproperty to validate actions (only increateoperations) to validate linked dataset propertiesTests included
(WIP pending tests and documentation)
Documentation
official documentation info
Docs are included in SciCatProject/documentation#56