Unit test case performance

[cyrille-artho]

With now a bit over 1000 unit tests, the CI runs start to take a bit more than just five minutes. Some tests appear slower than others. Without coverage information (see #417), we don't really know if we can easily drop a slow test. Some information on the slowest tests can help regardless, as they can perhaps be simplified or omitted.

[Tejasker]

hey @cyrille-artho is there anything that i can start as of now & improve the perfomance .
do let me know

Thanks

[cyrille-artho]

hey @cyrille-artho is there anything that i can start as of now & improve the perfomance . do let me know

Thanks

Hi,
To my knowledge, Gradle does not track the time per unit test, but it is easy to add this (temporarily) to the build setup:

https://stackoverflow.com/questions/41273942/how-to-get-test-timing-information-from-gradle-output-during-build

After you have added this, you can observe the time per unit test. I have tried this myself and saved the result in test-time.log; once you have the output, you can process it, for example, like this:

grep 'Total time.* was' test-time.log | sed -e 's/\(.*\) was \([0-9]*\)$/\2 \1/' | sort -rn | head -50

Looks at tests that take a rather long time to execute.
Are they perhaps using parameters/settings that are higher than necessary? Are there other ways how they might be simplified without losing coverage or fault-detection capabilities?

The 50 slowest tests (the output of the above command) are these:

13853 Total time of testProxyCreationInCaseOfChoiceGenerator
5037 Total time of testRecursiveJoinThreadGroup
2965 Total time of testStackWalkerInCaseOfChoiceGenerator
1509 Total time of testSimpleParse
1361 Total time of testResourceAcquisition
1303 Total time of testCountDown
1152 Total time of testFunctionality
893 Total time of testLambdaSerialization
877 Total time of testTimeoutExchange
804 Total time of testErrorTrace
706 Total time of testBasicProxy
626 Total time of testChildHandler
622 Total time of testCharBufferConstructor
618 Total time of testInterns
613 Total time of numOfThreadsTest
611 Total time of testBFSHeuristic
607 Total time of testPrintCharFormat
600 Total time of testDepth
588 Total time of testSetAndGetTimeZone
563 Total time of nativePeerTest
561 Total time of methodCloneTest
557 Total time of testMultipleIdleFinalizerThreads
557 Total time of testFormatWithTimeZone
542 Total time of systemClassLoaderTest
539 Total time of testStatelessRandomSearch
526 Total time of testShutdown
524 Total time of testGotoW
516 Total time of testYield
510 Total time of argTestNumber
504 Total time of testStopBlocked
502 Total time of testDeepStack
500 Total time of testWaitSyncInterrupt
500 Total time of testSimpleWait
499 Total time of testGetPackages
498 Total time of testSGOIDs
491 Total time of testGetResourceAsStream
488 Total time of testFuncObjAssignment
481 Total time of testWaitingSuspendNotifyResume
471 Total time of testStringConcatenationWithEmptyString
460 Total time of testSimpleJoin
454 Total time of outOfMemoryErrorFails
453 Total time of testInterfacePeer
453 Total time of testHighOrderRace
447 Total time of test
445 Total time of testArithmeticOps
442 Total time of ensureCompatibility
441 Total time of testAvailableProcessors
440 Total time of testNoConcurrentClinit
437 Total time of testBFSHeuristic
436 Total time of testThreadLocalWithInitial

Obviously, the three slowest tests add significantly to the execution time, so if there is something that can be simplified there, it will have an impact on the total time (and energy) used in future CI builds.

[cyrille-artho]

Note that a slow test is not necessarily overly complex; perhaps it simply tests a feature that is quite slow in JPF. In that case, it would be great if such a feature could be optimized, but this is usually easier said than done.

[Tejasker]

Hi @cyrille-artho, so based on your previous answer the way to increase the performance is to optimize the feature that takes a long time to test. so, In that case, we can boil it down to the classes & functions that are a bit slow in terms of execution & optimize those but as said earlier "this is usually easier said than done".

[cyrille-artho]

Yes, to some extent. For each slow test, there are various possible outcomes:

1. The test uses parameter settings that are too high or otherwise cause it to take too much time. A smaller number/input may result in a faster test. In this case, we still want to be confident that the quality of the test is not compromised by making it faster.
2. The test uses a fairly small parameter, but the operation is very slow. Perhaps this is a sign of a performance bug. Such a bug can be hard to fix.
3. The test is slow, but there is no obvious cause of why. In this case, we just leave it as is (pretty much the same outcome as for item 2).

The goal here is mostly to identify if some very slow tests fall into group 1; these are easy to "fix" by making their inputs smaller. The other cases are hard, but it can still be helpful to identify them, as this could be a future project (if a performance improvement seems feasible).

[Tejasker]

hi @cyrille-artho, so what if we can list out the methods that are taking longer time in terms of execution & also verify your point as such if those methods having input parameters that are quite large affecting the time taken during execution . If possible we can take up the 3rd point mentioned above and make necessary changes if you agree.

[cyrille-artho]

I think identifying and also simplifying tests that are "bigger" than necessary is the easiest way to find potential savings in test time. So I would try that first. Bottlenecks in JPF itself may be easy to identify, but they are usually hard to fix.

[Tejasker]

sure @cyrille-artho , I am willing to contribute if you require any sort of small hand in fixing those issue I am glad to join with you.
In the mean time I will try to study the code base of JPF & be ready for thsi

[harshal-rembhotkar]

Hi @cyrille-artho , can i work on this issue ? i want to contribute to this , i have a knowledge of Unit testing. i read above threads and understood the problem of this issue.

[cyrille-artho]

Yes, it seems that no one is working actively on this. However, I've made a quick check earlier, and it may not be possible to find an easy way to speed up the unit tests. I haven't looked at this in depth, though.