# robots.txt for sabaki.dance # # A crawler obeys exactly ONE group: the most specific `User-agent` that matches # it, and nothing from the others. So every group below has to repeat the # private-route list — an AI group that says only `Allow: /` is not inheriting # the restrictions from `*`, it is inviting the bot into /users and /messages. # # The disallowed prefixes are written WITHOUT a trailing slash on purpose: # `Disallow: /users/` does not match `/users`, which used to be crawlable for # exactly that reason. # # NEITHER `/auth` NOR `/account` is in this list, and the reason is the same # one: both are LINKED FROM PUBLIC HTML — `/account` sits in the header of # every page, and event pages link to `/auth/login?redirectPath=/event/…`. So # Google finds the URL no matter what this file says, and a `Disallow` then # only stops it from recrawling to SEE the noindex tag. The URL stays in the # index forever as "Indexed, though blocked by robots.txt" — which is exactly # what happened to `/account` (196 impressions, 11 clicks, Jun–Sep 2026) while # it was disallowed here. Both send `noindex, nofollow` from their layout # (`app/auth/layout.tsx`, `app/account/layout.tsx`) instead. # # The rule that follows: a private route may only be disallowed here as long as # nothing public links to it. Link to it, and it must be noindexed instead. # # That rule protects a search INDEX that can list a URL it may not fetch — # Google (`*`) and Bing. It does not protect the AI groups below, and applying # it there cost real crawl budget: every event page links a login, a register # and a Google-sign-in URL carrying its own `redirectPath`, so the catalog # multiplies into tens of thousands of distinct URLs with no content. Measured # 2026-09-29 on 50 000 AI-bot requests (Cloud Run logs, ~1.5 days): 7 024 were # /auth/login, /auth/register or /api/auth/google — Amazonbot 2 939, # OAI-SearchBot 1 176, ClaudeBot 1 003 (two thirds of its fetches), Meta 814. # So the AI groups disallow /auth, /account and /api; nothing there can be # cited, and /api/auth/google only redirects to Google's consent screen. # `*` and bingbot keep crawling them to see the noindex. # Search engines and everything else. Bing matters twice over: it is # also the index ChatGPT's search reaches for. User-agent: * Allow: / Disallow: /users Disallow: /messages Disallow: /notifications Disallow: /saved-events Disallow: /invite Crawl-delay: 1 # Bing / Copilot. Copilot has no crawler of its own: it cites Bing's index, # which bingbot builds. Named here (not left on `*`) so Bing does not inherit # Crawl-delay: 1 — the catalog is ~10 000 URLs and ChatGPT search reads this # index. MicrosoftPreview is the snapshot fetcher for Copilot/Bing cards. User-agent: bingbot User-agent: BingPreview User-agent: MicrosoftPreview Allow: / Disallow: /users Disallow: /messages Disallow: /notifications Disallow: /saved-events Disallow: /invite # --- AI / LLM crawlers (explicit allow) --- # Retrieval bots and training bots are both welcome on the public event # catalog: we want ChatGPT, Claude, Gemini, Perplexity and the rest to be # able to answer questions about Salsa / Bachata / Kizomba events with our # data, and to cite the event page when they do. # No Crawl-delay for these — the catalog is ~10 000 URLs and they should be # able to take it in one pass. Exception: ClaudeBot, see below. # OpenAI: training, search index, user-initiated fetch. User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User Allow: / Disallow: /users Disallow: /messages Disallow: /notifications Disallow: /saved-events Disallow: /invite Disallow: /auth Disallow: /account Disallow: /api # Anthropic training crawler. Measured 2026-10-12 (Cloud Run, 12:00-14:00 UTC): # 4 201 requests in 2 h (0 the day before), 2 539 of them on the /de/ pages; # human document latency p90 rose 288 -> 884 ms. Crawl-delay 5 = ~17 000 URLs/day, # which still covers the ~20 000-URL catalog (EN + /de) in ~1.2 days. Anthropic # documents that ClaudeBot honors Crawl-delay. Claude-Web and anthropic-ai are # the retired names, kept so an older crawler still matches a group that names it. User-agent: ClaudeBot User-agent: Claude-Web User-agent: anthropic-ai Allow: / Disallow: /users Disallow: /messages Disallow: /notifications Disallow: /saved-events Disallow: /invite Disallow: /auth Disallow: /account Disallow: /api Crawl-delay: 5 # Anthropic search index + user-initiated fetch: no delay, so Claude answers # keep fetching live. User-agent: Claude-SearchBot User-agent: Claude-User Allow: / Disallow: /users Disallow: /messages Disallow: /notifications Disallow: /saved-events Disallow: /invite Disallow: /auth Disallow: /account Disallow: /api # Google: Gemini training / grounding. Separate opt-in from Googlebot, # which is covered by `*` above and is what feeds AI Overviews. User-agent: Google-Extended User-agent: Google-CloudVertexBot Allow: / Disallow: /users Disallow: /messages Disallow: /notifications Disallow: /saved-events Disallow: /invite Disallow: /auth Disallow: /account Disallow: /api # Perplexity: index and user-initiated fetch. User-agent: PerplexityBot User-agent: Perplexity-User Allow: / Disallow: /users Disallow: /messages Disallow: /notifications Disallow: /saved-events Disallow: /invite Disallow: /auth Disallow: /account Disallow: /api # Apple Intelligence. Plain Applebot (Siri, Spotlight) falls under `*`. User-agent: Applebot-Extended Allow: / Disallow: /users Disallow: /messages Disallow: /notifications Disallow: /saved-events Disallow: /invite Disallow: /auth Disallow: /account Disallow: /api # Amazon: Alexa+ / Rufus. User-agent: Amazonbot User-agent: Amzn-SearchBot Allow: / Disallow: /users Disallow: /messages Disallow: /notifications Disallow: /saved-events Disallow: /invite Disallow: /auth Disallow: /account Disallow: /api # Other assistants. User-agent: MistralAI-User User-agent: DuckAssistBot User-agent: cohere-ai User-agent: YouBot Allow: / Disallow: /users Disallow: /messages Disallow: /notifications Disallow: /saved-events Disallow: /invite Disallow: /auth Disallow: /account Disallow: /api # Common Crawl — the training corpus behind many open models. User-agent: CCBot Allow: / Disallow: /users Disallow: /messages Disallow: /notifications Disallow: /saved-events Disallow: /invite Disallow: /auth Disallow: /account Disallow: /api # Meta AI: Meta-ExternalAgent collects pages for Meta's AI models and for # Meta AI answers inside WhatsApp, Instagram and Facebook. Deliberately allowed # (owner decision 2026-10-08: ~2-3 EUR/month of Cloud Run render cost, not worth # losing Meta AI in WhatsApp, our main channel). Link previews are fetched by # facebookexternalhit, which is not in this group and falls under `*`. User-agent: Meta-ExternalAgent User-agent: FacebookBot Allow: / Disallow: /users Disallow: /messages Disallow: /notifications Disallow: /saved-events Disallow: /invite Disallow: /auth Disallow: /account Disallow: /api # Sitemaps # The index; it points at /sitemaps/pages.xml, events-.xml, venues-.xml # and profiles-.xml, together covering the whole upcoming catalog. Sitemap: https://sabaki.dance/sitemap.xml