Add support for skipping nofollow links

[cipriancraciun]

Especially in relation to #78 (but this should apply independently of recursion), it would be nice to have a flag that would instruct lychee to just skip any <a> links that are marked with rel="nofollow", rel="...,nofollow,...", etc., as specified in the HTML standard (https://developer.mozilla.org/en-US/docs/Web/HTML/Attributes/rel).

The reasoning behind this, is say one has a site with lots of user-generated-content (wikis, comments, blogs, etc.), and the code already marks these "untrusted" links with nofollow. At the moment if one runs lychee on such a site, at best it will get loads of errors, and at worst it could actually be used to DOS the targeted sites. (Granted one could use exclusion, but that list would be so long and so often changing, that perhaps an --include-only option would be better.)

However with nofollow, the site administrator can still use lychee to check that important (and trusted) resources and links are still functional, while skipping those untrusted links in the user-generated-content.

[lebensterben]

yes. this definitely should be made default.

[mre]

Sounds good. We should add support for it.
In case anyone stumbles upon this in the future, there is an --include option as a workaround for now:

--include <include>... URLs to check (supports regex). Has preference over all excludes

[mre]

@cipriancraciun, thanks for the suggestion. Please check out #572 if you find the time.