[INTEGRATION_TESTS] Latency before effective tests starts

[echarles]

On my env, once the minikube is launched, I have to wait some minutes before the first test method starts. 1 CPU is used during that time by maven of VBox.

Is this expected or do I have to fix something on my env?

...
Creating a generic function for ‘atan2’ from package ‘base’ in package ‘SparkR’
Creating a generic function for ‘ifelse’ from package ‘base’ in package ‘SparkR’
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (SparkR)

==> Long time (+/-10 minutes, it depends)

- Run PySpark Job on file from SUBMITTER with --py-files
...

[ifilonenko]

This is because you are building docker images with the new build of the spark distribution that the images are using. If you are merely modifying the integration tests you can go into the integration tests folder and comment out the docker build logic as you would just reuse existing images. The time to bring up the mini kube instance is also a bottle-neck. However we have a PR that is addressing this, by persisting the minkube

[echarles]

Thx @ifilonenko

I guess you are referring to the SparkDockerImageBuilder scala class which should be commented.

Are you referring to the spark.docker.test.persistMinikube property which is already in the source? If not, can you point the PR you are mentioning to seep things on my side?

[ifilonenko]

#521

[ifilonenko]

Can this be closed?

[echarles]

closed