User-agent: * Allow: / Disallow: /app/ Disallow: /internal/ Disallow: /api/ Disallow: /portal/ Disallow: /portal.html Disallow: /admin/ Disallow: /login Allow: /app/login Allow: /signup # NOTE: do NOT re-add `Disallow: /*?utm_*` or `Disallow: /*?ref=*`. # Every campaign link we publish (LinkedIn, email signatures, GBP, the # webcallhub.fi → .com signup hop) carries those params. Blocking them stops # Google reading the page's rel=canonical, so instead of folding the params # onto the clean URL it files ~26 URLs under "Blocked by robots.txt" and # drops their link signal. The self-referencing canonical on every page # already does the consolidation correctly — robots.txt only gets in its way. # AI / LLM crawlers — explicitly allow so WebCallHub appears in AI answer engines User-agent: GPTBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup User-agent: ClaudeBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup User-agent: Google-Extended Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup User-agent: PerplexityBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup User-agent: CCBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup User-agent: anthropic-ai Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup # AI search / answer-engine crawlers (SearchGPT, ChatGPT browse, Claude, Perplexity) User-agent: OAI-SearchBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup User-agent: ChatGPT-User Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup User-agent: Claude-SearchBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup User-agent: Claude-User Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup User-agent: Perplexity-User Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup # Apple Intelligence / Siri User-agent: Applebot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup User-agent: Applebot-Extended Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup # Meta AI User-agent: Meta-ExternalAgent Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup # FacebookBot — Meta's AI training/retrieval crawler for Meta AI, distinct # from Meta-ExternalAgent/Meta-ExternalFetcher; added 2026-09-28 after it was # reported crawling at GPTBot-level volume since March 2026. User-agent: FacebookBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup User-agent: Meta-ExternalFetcher Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup # Amazon Alexa / Rufus User-agent: Amazonbot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup # DuckDuckGo AI answers User-agent: DuckAssistBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup # Mistral Le Chat User-agent: MistralAI-User Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup # ByteDance / Doubao User-agent: Bytespider Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup # You.com User-agent: YouBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup # Cohere User-agent: cohere-training-data-crawler Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup # Block aggressive content scrapers that don't drive traffic User-agent: SemrushBot Crawl-delay: 10 Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup User-agent: AhrefsBot Crawl-delay: 10 Disallow: /admin/ Disallow: /api/ Disallow: /app/ Disallow: /internal/ Disallow: /login Disallow: /portal/ Disallow: /portal.html Allow: /app/login Allow: /signup Sitemap: https://webcallhub.com/sitemap.xml # AI product information (llms.txt standard) # https://webcallhub.com/llms.txt — summary # https://webcallhub.com/llms-full.txt — detailed product info