Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement Export for Job Framework #1382

Merged
merged 15 commits into from
Oct 18, 2024
Merged

Implement Export for Job Framework #1382

merged 15 commits into from
Oct 18, 2024

Conversation

roegi
Copy link
Contributor

@roegi roegi commented Oct 18, 2024

  • Enhance RunEmrJob-step to run locally
  • Some fixes on SQLQuery related to quoting and query-context
  • Refactor existing parts of import functionality to make it re-usable for export implementation
  • Implement new Step for Job Framework "ExportSpaceToFiles"
  • Add compiler implementation for jobs that export space data
  • Streamline Ref parsing & validation and improve error messages
  • Some minor refactorings on dependent connector resources
  • Add Dockerfile / update docker compose yml to run Job Framework in Docker
  • Add further Job Framework tests and make them runnable within Github pipeline
  • Fully deactivate a space during import (if indices are dropped) to prevent read access without indices

roegi and others added 15 commits October 18, 2024 16:34
@roegi
Streamline implementation of PSQLXyzConnector
5577b50 
- When errors occur, do not return them in the process methods anymore, but rather throw them
- Use actual return types for all the process methods of all event types
- Fix IterateChangesets & IterateChangesetsEvent to only expect one response type rather than producing two different ambiguous types
  - Adjust ChangesetApi, ChangesetCollection, Changeset accordingly
- Remove testing code from actual PSQLXyzConnector implementation and move it into the TestStorageConnector
  - Update related test (PropertiesSearchIT) accordingly
- Replace boxed java types by their scalar types at several places
- Move some of the validation implementations from PSQLXyzConnector into the according QueryRunner implementations
- Fix ModifySubscription QR to not misuse a FeatureCollection as response vehicle, but rather use a SuccessResponse instead

Signed-off-by: Benjamin Rögner <[email protected]>
@roegi
Remove usage of apache-commons lib in xyz-psql-connector module
e485634 
Signed-off-by: Benjamin Rögner <[email protected]>
@roegi
Streamline Ref parsing & validation and improve error messages
d420610 
Signed-off-by: Benjamin Rögner <[email protected]>
@roegi
Move export workarounds into ExportSpace query runners
c95fca7 
Signed-off-by: Benjamin Rögner <[email protected]>
@roegi
Add QueryBuilder impl for GetFeaturesByGeometry
7821616 
Signed-off-by: Benjamin Rögner <[email protected]>
@mchrza @roegi
- Add first implementation of ExportSpaceToFiles step
d4d6a7e 
- Set scriptResourcePath inside new constructor of DatabaseSettings. If the scriptResourcePath is not set - checkScripts() will do nothing
- Move database script installation to DatabaseSettings
- Move Runtime-package to tools module + rename
- Rename scripts. Add transport.sql
- Some smaller refactorings regarding ImportFilesToSpace step
- Create new TestClasses for different Steps
- Align reading space()
- Add first implementation of ExportToFiles Compiler
- Create new scripts for export, import, transport
- Centralize logging in TransportTools
- Move import related functions from ext.sql
- Move space related fields to SpaceBasedStep.

Signed-off-by: Maximilian Chrzan <[email protected]>
Signed-off-by: mchrza <[email protected]>
@roegi
Rename files of FileStatistics
c6bfd2e 
Signed-off-by: Maximilian Chrzan <[email protected]>
Signed-off-by: mchrza <[email protected]>
Signed-off-by: Benjamin Rögner <[email protected]>
@mchrza @roegi
- Add local implementation to RunEmrStep
bb78cd4 
- Some minor cleanups
- Use simulatedLambda Requests for Tests.
- Prepare usage of QueryBuilder.
- Change to PropertiesQuery Object in Filters.
- Move SpatialFilter to step module.
- Shift parse logic from ApiParam to PropertiesQuery class.

Signed-off-by: Maximilian Chrzan <[email protected]>
Signed-off-by: mchrza <[email protected]>
@roegi
- Adjust NaN replacement query-fragment to set case-insensitivity fla…
22264e6 
…g using the "flags" param of regexp_replace()

  - That prevents confusion with the question marks sign to be interpreted as query-parameter placeholder
- Add implementation for DownloadUrl#store()
- Simplify EMR step handling in GraphTransformer
- Add default implementation for LambdaBasedStep#init() so that only when needed it must be overridden by subclasses
- Use standard method #registerOutputs() in RunEmrJob to upload the generated files
- Fix RunEmrJob#validate() method to not throw a validation exception for missing input files but just report not being ready instead
- Undo renaming of Step RunEmrJob
- Enhance ExportSpaceToFiles step to use GetFeaturesByGeometryBuilder
- Add new method SQLQuery#toExecutableQueryString()

Signed-off-by: Benjamin Rögner <[email protected]>
@mchrza @roegi
- Fix flickering test
74748a9 
- Fix GeometryValidation in case of h3Index and referenced Features
- Add job-service docker file
- Add handling of versionRef to export step (resolve & validate)
- Add missing forward of filters, context and versionRef to ExportCompiler
- In tests, add possibility to createSpace with spaceDefinition
- Add Tests for GeometryValidator
- Add function to retrieve the used "searchableProperties"
- Add geometry validation using new GeometryValidator to centralize related code
- Fix bug in GetFeaturesByGeometry QR that caused H3-index tests to fail
- Add first IntegrationTests
- Several improvements
- Fix bug in SQLQuery that for fully substituted (executable) queries, in some scenarios the custom quotes were not "escaped"
  - Introduce a new "escaping" mechanism for custom quotes
- Fix two bugs which have happened during job-deletions.
- Fix bug in SQLQuery that values could not be used to replace parameter placeholders if containing a dollar sign
- Fix bug in GetFeaturesByGeometryBuilder (missing AND)
- Set threadCondition filter fragment in all sub-parts of the content query correctly
- Fix query building for "GetFeaturesByGeometry" queries if geometry or ref is null

Signed-off-by: Benjamin Rögner <[email protected]>
Signed-off-by: Maximilian Chrzan <[email protected]>
Signed-off-by: mchrza <[email protected]>
@roegi
Fully deactivate a space during import (if indices are dropped) to pr…
9745d38 
…event read access without indices

Signed-off-by: Benjamin Rögner <[email protected]>
@mujammil10 @roegi
Fix s3 path for local EMR
70588ca 
Signed-off-by: Ansari, Mujammil <[email protected]>
@mchrza @roegi
- Fix GitHub testing - Remove interactive terminal mode
38d9d04 
- Enable tests
- Add JobService setup run
- Upload generated files to provided path instead of registering outputs.

Signed-off-by: Maximilian Chrzan <[email protected]>
Signed-off-by: mchrza <[email protected]>
@roegi
Simplify script installation / make it fully transparent to users of …
9d32c2a 
…DatabaseSettings

Signed-off-by: Benjamin Rögner <[email protected]>
@roegi
Fix rebase-conflicts regarding ChangesetApi updates
7d76c08 
Signed-off-by: Benjamin Rögner <[email protected]>
@roegi roegi requested a review from a team as a code owner October 18, 2024 16:34
@roegi roegi merged commit 5da046d into master Oct 18, 2024
3 checks passed
@roegi roegi deleted the jobRefac7 branch October 18, 2024 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@roegi @mchrza @mujammil10