Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use asyncio to gain performance #162

Open
netomi opened this issue Nov 28, 2023 · 2 comments · May be fixed by #163
Open

Use asyncio to gain performance #162

netomi opened this issue Nov 28, 2023 · 2 comments · May be fixed by #163

Comments

@netomi
Copy link

netomi commented Nov 28, 2023

I am using python-inspector as part of the oss review toolkit and I noticed that the analysis of some python projects is rather slow though it has a poetry.lock file and thus the resolution should be straight-forward.

Some debugging revealed that most of the time is spent in downloading stuff from pypi. I could greatly speed this up using async io requests using the aiohttp module. Additionally, I would suggest to replace the existing cache with aiohttp-client-cache.

I will prepare a PR to showcase the use of async io and how you can gain considerable speedups using that approach by parallelizing the downloads from pypi instead of doing them sequentially which take a while if you have a larger number of dependencies.
@netomi
Copy link
Author

netomi commented Nov 28, 2023

Using the current version in my project results in this:

tn@proteus:~/workspace/netomi/ort$ time python-inspector --python-version 311 --operating-system linux --json-pdt /tmp/ort-PythonInspector676305276266227952/python-inspector16065407924165681261.json --analyze-setup-py-insecurely --requirement /tmp/ort-Poetry6870289546925138000/requirements.txt9586036788288994416.tmp 

real	0m29,104s
user	0m12,745s
sys	0m0,353s

Using aiohttp:

tn@proteus:~/workspace/netomi/python-inspector$ time /home/tn/workspace/netomi/python-inspector/venv/bin/python /home/tn/workspace/netomi/python-inspector/src/python_inspector/resolve_cli.py --operating-system linux --python-version 311 --json-pdt /tmp/ort-PythonInspector676305276266227952/python-inspector16065407924165681261.json --analyze-setup-py-insecurely --requirement /tmp/ort-Poetry6870289546925138000/requirements.txt9586036788288994416.tmp 

real	0m16,893s
user	0m8,284s
sys	0m0,439s

@netomi netomi linked a pull request Nov 29, 2023 that will close this issue
Open
@TG1999
Copy link
Contributor

TG1999 commented Nov 30, 2023 •
edited
Loading

Thanks @netomi for raising this issue and your PR for fixing this : )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use async io where possible to improve runtime performance netomi/python-inspector
2 participants
@netomi @TG1999