Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invalid robots.txt file #70

Open
smolgeorgie opened this issue Feb 5, 2025 · 0 comments
Open

Invalid robots.txt file #70

smolgeorgie opened this issue Feb 5, 2025 · 0 comments

Comments

@smolgeorgie
Copy link

smolgeorgie commented Feb 5, 2025

Robots.txt file consists of user-agent and the type of access.

This access is now:
Allow: *

* is an invalid path for this syntax.

The solution:

If you want to deny all access, you use: Disallow: /
If you want to give all access, you use: Disallow: - As you can see, there is no path, its blank. This blank means that there's nothing to disallow. Repeating: There is nothing behind the :
If you want to deny access to certain files, you use: Disallow: [path]

Side note:

You can use Allow, but I quote: ''This command makes it possible to allow bots to reach one particular webpage, while disallowing the rest of the webpages in the file. Not all search engines recognize this command.''

https://www.cloudflare.com/learning/bots/what-is-robots-txt/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: No status
Development

No branches or pull requests

1 participant