Be nice on the web

Makes it easy to add robots.txt, sitemap and web app manifest during build to your Astro app.

A native Rust port of Google's robots.txt parser and matcher C++ library.

R 📦 for parsing and checking robots.txt files 🤖

Generator robots.txt for node js

A webpack plugin to generate a robots.txt file

The right robots.txt file for your project

Alternative robots parser module for Python

Go robots.txt parser

data.txt: Specification

A sane, minimal robots.txt file (for the western world)

Go language library for parsing Sitemaps

This is a small package to make parsing robots.txt rules easier.

Workshop materials for creating the Center City Sips District 2022 interactive map

📚 Libraries for retrieving and parsing robots.txt & sitemaps.

data.txt: Data for the Web

⚙️ A quality `robots.txt` ruleset parser to ensure your application follows the standard specification for the file.

Parse robots.txt and traverse sitemaps.

Replaces the default virtual robots.txt generated by WordPress with an editable one, and deletes any physical robots.txt file that may already exist.

The site seo scanner.