A python class to facilitate compliance with the crawl-delay directive in a robots.txt file
-
Notifications
You must be signed in to change notification settings - Fork 0
A python class to facilitate compliance with the crawl-delay directive in a robots.txt file
License
ebeckm/Crawl_delay
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
A python class to facilitate compliance with the crawl-delay directive in a robots.txt file
Resources
License
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published