Wappalyzer - crawling of resources on a website

[stephanie0x00]

Describe the bug
The very first wappalyzer performed some kind of crawling on websites to find software and it's versions. However this version was outdated and didn't provide any proof of how the data was gathered for reproducibility. The newer versions of our wappalyzer implementation do not crawl all page of a website, thus software instances and it's versions are often missed, as these are often found in external files (e.g. https://example.com/js/core.js contains the string jQuery 1.2.3).

Issue identified with #3800.

The question now is:

• How are we going to crawl the resources?
• What depth are we going to crawl?
• What scopes?
• Others?

Expected behavior
Some kind of crawling should be performed to identify these files and check for software and software versions.

OpenKAT version
main