# Common Crawl User-agent: CCBot Disallow: / # GPT User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / # Global User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php