Speedup test runner

[ErikSchierboom]

The current test runner is very slow when running via the Docker container: 20s for a simple hello-world exercise, whereas running it locally without Docker has this happen in 2/3s. With some digging, I found that the likely culprit is the Gradle daemon. If I disable the Gradle daemon on my local system and removing any previous build artifacts, running the tests becomes a lot slower: 8/9s.

I thus set out to try to enable the Gradle daemon in the Docker build by adding the following line in the Dockerfile:

RUN gradle

This starts the Gradle daemon, but unfortunately, somehow this daemon is then stopped before it can be used by the test runner's bin/run.sh script. This can be verified by running gradle --status in bin/run.sh:

cd $2 # Solution directory
gradle --status

which outputs:

No Gradle daemons are running.
   PID STATUS   INFO
    43 STOPPED  (by user or operating system)

Only Daemons for the current Gradle version are displayed. See https://docs.gradle.org/6.8.3/userguide/gradle_daemon.html#sec:status

It would be great if someone could figure out how to re-use the Gradle daemon.

There might also be other issues slowing compilation down. If you know any, feel free to test them out.

I've experimented a bit with all this in this branch.

[ErikSchierboom]

I've also tried a different approach where I ran kotlinc manually. Unfortunately, that didn't appear to be any quicker. You can see this experiment in this branch.

[ErikSchierboom]

Whilst converting to Maven has helped, it still is quite slow. I've tried a number of different things:

• Changing the max heap size
• Upgrading maven
• Using kotlinc directly
• Using the new K2 kotlinc

erik@schierheim:~/exercism/kotlin-test-runner/tests/example-success$ time kotlinc src/main/kotlin/Leap.kt 

real    0m2.437s
user    0m5.924s
sys     0m0.528s

erik@schierheim:~/exercism/kotlin-test-runner/tests/example-success$ time kotlinc -language-version 2.0 src/main/kotlin/Leap.kt 
warning: language version 2.0 is experimental, there are no backwards compatibility guarantees for new language and library features

real    0m2.444s
user    0m5.581s
sys     0m0.659s

Unfortunately, none of this has any significant effect.

[ErikSchierboom]

I've also tried parallellizing surefire but that also didn't help:

<configuration>
  <parallel>all</parallel>
  <useUnlimitedThreads>true</useUnlimitedThreads>
</configuration>

[ErikSchierboom]

I've also tried to use the Maven Daemon, but no luck.

[ErwinOlie]

Maybe it's an unconventional idea. But what about not using an automated buildtool at all? Just compiling the main and test class will only take a few seconds at maximum. We don't have a complex dependency structure and I expect it to stay this way, so we can just wget the 2 or 3 jar's we need from a maven repo and then run kotlinc / java -jar. Is it worth my time to try to make a proof of concept, or is the idea a no-go?

[sanderploegsma]

It’s definitely worth investigating as we currently take a similar approach in the Java test runner: the solution and test files are compiled directly by the runner using the Java compiler API, and this indeed shaves off a bit of overhead.

Presumably a runtime Kotlin compiler can unlock something similar here. In fact, there’s a thread on the forum where I suggested something like this, feel free to weigh in there too: https://forum.exercism.org/t/kotlin-exercises-keep-timing-out/6891/16

[ErikSchierboom]

That sounds like a very valuable poc!
I've played with running the kotlin compiler via kotlinc but I didn't get much performance benefits. But doing it in memory could be a lot better

[ErwinOlie]

I did some experiments using the HelloWorld example.

First I looked at the current structures, put everything in the same directory with the pre-downloaded dependency jars from Maven Central.

kotlinc HelloWorld.kt -d HelloWorld.jar
kotlinc HelloWorldTest.kt -cp junit-4.12.jar:HelloWorld.jar -d HelloWorldTest.jar
java -cp HelloWorldTest.jar:junit-4.12.jar:HelloWorld.jar:hamcrest-core-1.3.jar org.junit.runner.JUnitCore HelloWorldTest

Found a random 3 year old docker image including the kotlin toolchain and OpenJDK12: zenika/kotlin:latest. I executed the commands inside the container on the mounted project files.

Results after execution:

04.21 sec
04.42 sec
08.39 sec
04.16 sec
04.65 sec

Because I wasn't satisfied with the times, and because the reporting is hard to use from junit4 without a Surefire plugin or similar I also tried an upgrade to junit5. And did the same but with the following commands:

kotlinc HelloWorld.kt HelloWorldTest.kt -cp junit-jupiter-api-5.10.0.jar -d HelloWorldTest.jar
java -jar junit-platform-console-standalone-1.10.0.jar -cp HelloWorld.jar:HelloWorldTest.jar --scan-class-path

Results after execution:

02.71 sec
02.72 sec
02.83 sec
03.35 sec
02.73 sec

The times improved and junit5 has decent built-in test reports.

These times describe running a docker container, compiling the main class and testclass, executing the test, and generating a test report. What's missing from the equation is the overhead of running the autotest-runner.jar. What's also missing is potential further improvements by using the latest kotlin compiler and jdk versions.

I think exploring this pathway is worth a good try, but junit5 seems to be preconditions. I'll have a look next week if I can contribute the junit update.

I also did a quick research in compiling and dynamically loading the classes, which seems to be even more promissing but is also a bit harder to execute. For that the junit5 update and parsing of the new report format seems to be preconditions as well so I think this can be a step in the right direction anyways.

Not sure if I've missed something important

[ErikSchierboom]

That all sounds very hopeful!

I think exploring this pathway is worth a good try, but junit5 seems to be preconditions. I'll have a look next week if I can contribute the junit update.

Great!

[sanderploegsma]

@ErwinOlie an alternative approach to parsing the JUnit test report is to invoke JUnit dynamically from the test-runner, where you can pass it a TestExecutionListener. That way you are not restricted to the information that is put in the test report by JUnit, and it saves the overhead of invoking two different JARs on the JVM (if there is any).

Here's how we do it in the Java test runner, for example: https://github.com/exercism/java-test-runner/blob/main/lib/src/main/java/com/exercism/junit/JUnitTestRunner.java