-
Notifications
You must be signed in to change notification settings - Fork 1
herrbischoff/robots.txt
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
A sane, minimal robots.txt file (for the western world) You're unlikely to need any other crawlers on your site to be found. If you have special requirements or need/want to use local search engines, this is probably more of a starting point for you. As this blocks all but the excepted user agents, it should plainly identify crawlers that do not respect the settings.
About
A sane, minimal robots.txt file (for the western world)