Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some Storer and Reader functions could be parallelized #9

Open
iosonopersia opened this issue Apr 21, 2021 · 0 comments
Open

Some Storer and Reader functions could be parallelized #9

iosonopersia opened this issue Apr 21, 2021 · 0 comments
Labels
enhancement New feature or request

Comments

@iosonopersia
Copy link
Collaborator

Functions from Storer and Reader that execute SPARQL queries (SELECT/UPDATE) or that access RDF files (read/write), could be parallelized in some way to achieve better performance.

Generally speaking, applications that make use of oc_ocdm often need to import/store/upload a large quantity of data. This usually consumes a lot of time, which often represents the biggest percentage of total execution time.

An initial optimization approach can be found in Storer.upload_all(...), where the total amount of UPDATE queries to be performed is grouped into batches of configurable maximum size (see batch_size parameter).

@iosonopersia iosonopersia added the enhancement New feature or request label Apr 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant