---
name: robots-txt-ai-rules
description: Declare AI crawler rules in robots.txt
---

# Fix: robotsTxtAiRules

**Probe:** Parse robots.txt for known AI UAs (GPTBot, Claude-Web, PerplexityBot, CCBot, Google-Extended, Bytespider, …) or a `User-agent: *` group with rules.

**Add** explicit groups (or keep a clear `User-agent: *` policy):

```
User-agent: GPTBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: PerplexityBot
Allow: /
```

Spec: https://www.rfc-editor.org/rfc/rfc9309.html