# General crawl rules User-agent: * Disallow: /wp-admin/ Allow: /llms.txt Allow: /wp-admin/admin-ajax.php Disallow: /wp-includes/ Allow: /wp-includes/js/ Allow: /wp-includes/css/ Allow: /wp-includes/images/ Disallow: /trackback/ Disallow: /wp-login.php Disallow: /wp-register.php Disallow: /tag/* Disallow: /author/* Disallow: */?s* Disallow: */page/* # Search indexes and user fetches: allow AI search & user features User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User Allow: / # Training only: block AI foundation model scrapers User-agent: GPTBot User-agent: ClaudeBot Disallow: / # Sitemaps Sitemap: https://www.rydo.com.au/sitemap_index.xml