Towards better python support

[prabhu]

• cdxgen fails to look at all dependencies for python project #336
• scanning setup.py doesn't work if "install_requires" is initialized to a variable instead of a list of strings #337
• Support for huggingface models and datasets #338
• Parsing license strings in python bdist files is causing validation failures #186
• Add Dependencies in the BOM #125
• Python pyproject.toml support #349

The gist is that we need to parse Python code and identify import statements (including magic imports) to improve the accuracy for direct dependencies. This seems to be the approach taken by projects like pigar and pipreqs and my personal dep-scan.

Once we have the direct dependencies nailed, we can identify the indirect dependencies by querying pypi, a feature recently removed from cdxgen due to some bugs.

Perhaps, this is a great excuse to introduce atom from AppThreat to the CycloneDX community. It can parse Python and create a json slice for the Usages. We need an npm wrapper package to bundle atom as an optional dependency.

[prabhu]

Branch to follow - #347

Python is turning out to be one of the most complex feature developed for cdxgen. There is a lack of understanding on the following topics:

• What is a good logic to determining direct and indirect dependencies? With atom, we are parsing the application source and any setup.py to identify the imported modules. Then have a big mapping file between modules names and PyPI package names since these two could be different. This kinda gives us a list of direct dependencies but not the indirect dependencies. Should we get the indirect dependencies from PyPI? But then onto problem 2.
• What is a good logic to mimic the pip solver to determine the versions? cdxgen tries to create a virtual environment and then do a pip freeze to determine the list for requirements.txt projects. If creating virtual environments fail for any reason (due to lack of virtualenv package or other os errors) or for those projects with a complex setup.py file, we are stuck with a list without version numbers. While cdxgen then tries to find the version number by checking with PyPI this always gets us the latest version and not the version that pip might have resolved.

Some sample projects suggested by @Kasyap-R are scipy, requests

[prabhu]

cdxgen 8.6.0 got released with several improvements for Python. We focused on improving the precision and recall for direct dependencies first. The release was only possible with several community members' hard work.

In particular, I want to thank:

@Kasyap-R - For huge help from raising tickets with test cases to continuously testing our development branch
@DavidBakerEffendi - For the inspiration and for help with the creation of atom, which powers the new detection capabilities

To give you an idea about the level of improvements, let's take a complex project like scipy and look at the dependency list generated by GitHub.

https://github.com/scipy/scipy/network/dependencies

Now compare with the list from cdxgen

https://github.com/CycloneDX/cdxgen/actions/runs/5346454398/jobs/9693518561#step:13:606

We can continue to iterate and resolve any issues from this release and then look into improving the detection of transitive dependencies.

[prabhu]

In 9.1.x, we added transitive dependencies and dependency tree generation for setup.py, requirements.txt, and pyproject.toml projects.