Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migrate parameterized DroidBench tests from JUnit 4 to JUnit 5 #1298

Merged
merged 1 commit into from
Jul 31, 2023
Merged

Migrate parameterized DroidBench tests from JUnit 4 to JUnit 5 #1298

merged 1 commit into from
Jul 31, 2023

Conversation

liblit
Copy link
Contributor

@liblit liblit commented Jul 26, 2023

IntelliJ IDEA does not know how to migrate parameterized tests, so these test classes were migrated by hand.

@liblit liblit self-assigned this Jul 26, 2023
@liblit liblit marked this pull request as ready for review July 26, 2023 01:56
@liblit liblit enabled auto-merge (rebase) July 26, 2023 02:00
@github-actions
Copy link

github-actions bot commented Jul 26, 2023 •
edited
Loading

Test Results

   451 files  ±    0     451 suites  ±0   2h 48m 2s ⏱️ - 1m 40s
   730 tests  - 116     713 ✔️  - 116  17 💤 ±0  0 ❌ ±0 
2 804 runs  ±    0  2 734 ✔️ ±    0  70 💤 ±0  0 ❌ ±0 

Results for commit ab2f5ee. ± Comparison against base commit 70eb6e9.

This pull request removes 232 and adds 116 tests. Note that renamed tests count towards both. 
com.ibm.wala.dalvik.test.callGraph.droidbench.AliasingTest ‑ runTest[DroidBench: /Users/runner/work/WALA/WALA/dalvik/build/DroidBench/apk/Aliasing/Merge1.apk]
com.ibm.wala.dalvik.test.callGraph.droidbench.AliasingTest ‑ runTest[DroidBench: /home/runner/work/WALA/WALA/dalvik/build/DroidBench/apk/Aliasing/Merge1.apk]
com.ibm.wala.dalvik.test.callGraph.droidbench.AndroidSpecificTest ‑ runTest[DroidBench: /Users/runner/work/WALA/WALA/dalvik/build/DroidBench/apk/AndroidSpecific/ApplicationModeling1.apk]
com.ibm.wala.dalvik.test.callGraph.droidbench.AndroidSpecificTest ‑ runTest[DroidBench: /Users/runner/work/WALA/WALA/dalvik/build/DroidBench/apk/AndroidSpecific/DirectLeak1.apk]
com.ibm.wala.dalvik.test.callGraph.droidbench.AndroidSpecificTest ‑ runTest[DroidBench: /Users/runner/work/WALA/WALA/dalvik/build/DroidBench/apk/AndroidSpecific/InactiveActivity.apk]
com.ibm.wala.dalvik.test.callGraph.droidbench.AndroidSpecificTest ‑ runTest[DroidBench: /Users/runner/work/WALA/WALA/dalvik/build/DroidBench/apk/AndroidSpecific/Library2.apk]
com.ibm.wala.dalvik.test.callGraph.droidbench.AndroidSpecificTest ‑ runTest[DroidBench: /Users/runner/work/WALA/WALA/dalvik/build/DroidBench/apk/AndroidSpecific/LogNoLeak.apk]
com.ibm.wala.dalvik.test.callGraph.droidbench.AndroidSpecificTest ‑ runTest[DroidBench: /Users/runner/work/WALA/WALA/dalvik/build/DroidBench/apk/AndroidSpecific/Obfuscation1.apk]
com.ibm.wala.dalvik.test.callGraph.droidbench.AndroidSpecificTest ‑ runTest[DroidBench: /Users/runner/work/WALA/WALA/dalvik/build/DroidBench/apk/AndroidSpecific/PrivateDataLeak1.apk]
com.ibm.wala.dalvik.test.callGraph.droidbench.AndroidSpecificTest ‑ runTest[DroidBench: /Users/runner/work/WALA/WALA/dalvik/build/DroidBench/apk/AndroidSpecific/PrivateDataLeak2.apk]
…

com.ibm.wala.dalvik.test.callGraph.droidbench.AliasingTest ‑ [1] Aliasing/Merge1.apk
com.ibm.wala.dalvik.test.callGraph.droidbench.AndroidSpecificTest ‑ [10] AndroidSpecific/PublicAPIField1.apk
com.ibm.wala.dalvik.test.callGraph.droidbench.AndroidSpecificTest ‑ [11] AndroidSpecific/PublicAPIField2.apk
com.ibm.wala.dalvik.test.callGraph.droidbench.AndroidSpecificTest ‑ [1] AndroidSpecific/ApplicationModeling1.apk
com.ibm.wala.dalvik.test.callGraph.droidbench.AndroidSpecificTest ‑ [2] AndroidSpecific/DirectLeak1.apk
com.ibm.wala.dalvik.test.callGraph.droidbench.AndroidSpecificTest ‑ [3] AndroidSpecific/InactiveActivity.apk
com.ibm.wala.dalvik.test.callGraph.droidbench.AndroidSpecificTest ‑ [4] AndroidSpecific/Library2.apk
com.ibm.wala.dalvik.test.callGraph.droidbench.AndroidSpecificTest ‑ [5] AndroidSpecific/LogNoLeak.apk
com.ibm.wala.dalvik.test.callGraph.droidbench.AndroidSpecificTest ‑ [6] AndroidSpecific/Obfuscation1.apk
com.ibm.wala.dalvik.test.callGraph.droidbench.AndroidSpecificTest ‑ [7] AndroidSpecific/PrivateDataLeak1.apk
…

♻️ This comment has been updated with latest results.

@liblit liblit force-pushed the junit-5/migrate-parameterized branch from 3cd2490 to fd91301 Compare July 26, 2023 06:07
@msridhar
Copy link
Member

Do we expect the discrepancy between removed and added tests in the test results report?

@liblit
Copy link
Contributor Author

liblit commented Jul 26, 2023

Do we expect the discrepancy between removed and added tests in the test results report?

We do not. I'm still investigating, which is why I haven't requested your review yet. I'm glad to see that the new test summary report is serving as an extra check: exactly what I intended.

@liblit liblit force-pushed the junit-5/migrate-parameterized branch from fd91301 to e8f17dc Compare July 26, 2023 23:24
@liblit
Migrate parameterized DroidBench tests from JUnit 4 to JUnit 5
ab2f5ee 
IntelliJ IDEA does not know how to migrate parameterized tests, so
these test classes were migrated by hand.
@liblit liblit force-pushed the junit-5/migrate-parameterized branch from e8f17dc to ab2f5ee Compare July 27, 2023 14:18
@liblit
Copy link
Contributor Author

liblit commented Jul 31, 2023

In brief: we're not actually losing any tests in this PR. We're simply losing spurious variations in test names.

The test summary before this change reports 2,804 test runs and 846 tests. The 846 count is based on distinct test names. A single test name leads to multiple test runs because we test on multiple operating systems and Java versions.

The test summary after this change reports 2,804 test runs and 730 tests. So we actually have the same number of test runs across all tested platforms, but these test runs consist of fewer distinct test names.

A detailed, manual review of the before/after test names shows that several tests' before names included absolute path names, but are reported using shorter, relative path names after this change. Absolute paths vary by operating system, while the relative names do not. Thus, the before tests had 846 distinct names that reduced to just 730 distinct names after this change. To give one concrete example, tests using the com.ibm.wala.dalvik.test.callGraph.droidbench.ThreadingTest class include:

revision JDK OS Test Name
before 11 Linux runTest[DroidBench: /home/runner/work/WALA/WALA/dalvik/build/DroidBench/apk/Threading/Looper1.apk]
before 17 Linux runTest[DroidBench: /home/runner/work/WALA/WALA/dalvik/build/DroidBench/apk/Threading/Looper1.apk]
before 11 macOS runTest[DroidBench: /Users/runner/work/WALA/WALA/dalvik/build/DroidBench/apk/Threading/Looper1.apk]
after 11 Linux [5] Threading/Looper1.apk
after 17 Linux [5] Threading/Looper1.apk
after 11 macOS [5] Threading/Looper1.apk

@liblit liblit requested a review from msridhar July 31, 2023 01:28
msridhar
msridhar approved these changes Jul 31, 2023
View reviewed changes
Copy link
Member

@msridhar msridhar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the change and for digging in to check it!

@liblit liblit merged commit 4f39c61 into wala:master Jul 31, 2023
8 checks passed
@liblit liblit deleted the junit-5/migrate-parameterized branch July 31, 2023 01:38
@liblit
Copy link
Contributor Author

liblit commented Jul 31, 2023 •
edited
Loading

You're welcome, @msridhar! I'm really liking these “Test Results” summaries. We’re still getting used to them, but they’re going to be a nice piece of extra protection for us.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@msridhar msridhar msridhar approved these changes

Assignees

@liblit liblit

Labels
None yet
Projects
No open projects
Migrate from JUnit 4 to JUnit 5
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@liblit @msridhar