Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(npm): Remove unused parallelization constructs #9316

Merged
merged 2 commits into from
Oct 23, 2024

Conversation

fviernau
Copy link
Member

@fviernau fviernau commented Oct 22, 2024

The code looks like it would run things in parallel, but it executes getRemotePackageDetails() strictly sequentially. The pull request which had introduced these constructs 1 also introduced the caching. So, the speed-up observed back then was probably solely due to the added cache. Remove the parallelization constructs to not mislead the reader and to simplify.

Part of #9261.

@fviernau fviernau requested a review from a team as a code owner October 22, 2024 13:02
@fviernau fviernau force-pushed the npm-drop-fake-parallelization branch from 41c90e0 to 6f80410 Compare October 22, 2024 13:12
Copy link

codecov bot commented Oct 22, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 67.49%. Comparing base (8885a75) to head (697903b).
Report is 2 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #9316      +/-   ##
============================================
- Coverage     67.49%   67.49%   -0.01%     
- Complexity     1198     1200       +2     
============================================
  Files           241      241              
  Lines          8495     8493       -2     
  Branches        900      899       -1     
============================================
- Hits           5734     5732       -2     
  Misses         2399     2399              
  Partials        362      362              
Flag Coverage Δ
funTest-docker 60.54% <100.00%> (-0.05%) ⬇️
funTest-non-docker 33.63% <ø> (ø)
test 37.47% <0.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sschuberth
Copy link
Member

Mind having a look, @oheger-bosch?

Copy link
Member

@oheger-bosch oheger-bosch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes look good to me. Nice finding, that the async logic was rather useless.

@@ -24,11 +24,6 @@ package org.ossreviewtoolkit.plugins.packagemanagers.node
import java.io.File
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commit message:
Nit: "...strictly sequentially."
So, if I understand correctly, the whole fiddling with async had no real effect, since only a single package was processed, and the parsePackage function is only called sequentially? Maybe make this a bit more explicit in the commit message?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since only a single package was processed

hm, I made the observation when running the fun tests, so not a single package but multiple package were involved. (A single package cannot be processed in parallel, so I guess this is a mis-understanding)

and the parsePackage function is only called sequentially

I in fact verified that getRemotePackageDetails() executes only sequentally. But, parsePackage() is the only caller of that one, so I guess the same applied to parsePackage().

What change to the commit message do you suggest?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I understand, there are multiple aspects:

  • The withContext construct in getRemotePackageDetailsAsync is only exited after all coroutines started within it are finished. Therefore, the async invocations are useless, since the function waits until the asynchronous processing is complete.
  • But even on a higher level, no parallelization is applied; parsePackage is invoked sequentially for each package and waits for the result. So, even if asynchronous processing worked, it would not help here.

Do you agree with this?

Regarding the commit message, it is probably not necessary to describe this in detail. So, your text is probably fine.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you agree with this?

Yes, makes sense to me.

@@ -245,10 +245,6 @@ open class Npm(
* content via the `npm view` command. The result is a [Pair] with the raw identifier and the new package.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commit message nit: "...would have searched..."

@fviernau fviernau force-pushed the npm-drop-fake-parallelization branch 2 times, most recently from 783b8cc to cfdcadb Compare October 22, 2024 14:14
The code looks like it would run things in parallel, but it executes
`getRemotePackageDetails()` strictly sequentially. The pull request
which had introduced these constructs [1] also introduced the caching.
So, the speed-up observed back then was probably solely due to the
added cache. Remove the parallelization constructs to not mislead the
reader and to simplify.

[1]: #6009

Signed-off-by: Frank Viernau <[email protected]>
The log message sounds as if the function would have searched the
'package.json' and now found one. However, the function does not search
anything. Simply remove that log output to simplify an upcoming change.

Signed-off-by: Frank Viernau <[email protected]>
@fviernau fviernau force-pushed the npm-drop-fake-parallelization branch from 4dc3f13 to 697903b Compare October 23, 2024 09:44
@sschuberth sschuberth merged commit 7143c32 into main Oct 23, 2024
22 of 23 checks passed
@sschuberth sschuberth deleted the npm-drop-fake-parallelization branch October 23, 2024 10:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants