robots.txt, sitemap & security.txt config generator

Generate robots.txt, sitemap.xml and an RFC 9116 security.txt for a site: crawl rules, an optional AI-crawler opt-out, and a vulnerability-contact file.

All generators

Guides

• Nginx Reverse Proxy Configuration
• Caddy Reverse Proxy Configuration
• Nginx Static Site Configuration
• Caddy Static Site Configuration
• Nginx + PHP-FPM Configuration
• Caddy + PHP-FPM Configuration
• Nginx SSL/TLS Configuration
• Nginx Load Balancer Configuration
• Caddy Load Balancer Configuration
• Nginx Rate Limiting Configuration
• Nginx Security Headers Configuration
• Caddy Security Headers Configuration
• Nginx Gzip Compression Configuration
• Nginx Caching Configuration
• Caddy HTTPS Redirect Configuration
• Nginx WebSocket Proxy Configuration
• Nginx Content-Security-Policy Header
• Block Bad Bots and Scanners in Nginx
• fail2ban for Nginx
• nginx.conf Best Practices
• Nginx Location Regex Tester
• Deploy Nginx Config with GitHub Actions
• Nginx CORS Headers Configuration
• Nginx Cache-Control Headers Configuration
• Nginx 301 Redirect Map Configuration
• robots.txt, Sitemap and security.txt
• Apache .htaccess Configuration
• Docker Compose and Dockerfile Configuration
• systemd Service and Timer Configuration
• Cron Expression Explainer
• SSH Hardening with sshd_config
• Linux Firewall Rules: ufw, nftables and iptables
• Glob Pattern Tester
• Analysing Nginx Access Logs with awk