# robots.txt for melaya.org # # Public marketing surface (six products, equal weight): # / landing # /home explicit landing route # /product/agentic-framework Melaya Agents (visual agent builder) # /product/assistant Melaya Assistant # /product/agentic-device-control Melaya Device Control (mobile agents) # /product/agentic-browser-control Melaya Browser Control # /product/mcp Melaya MCP Server # /product/marketing Melaya Marketing # /product/agentic-trading-crew Melaya Labs trading crew (coming later) # /product/trading-engine Melaya Labs Rust engine (coming later) # /enterprise-ai-agents, /browser-ai-agents, /mobile-ai-agents topic landing pages # /use-cases, /use-cases/{slug} hub + 23 role/industry pages # /vs, /vs/{competitor} comparison hub + competitor pages # /guides/{slug} how-to guides # /blog, /blog/{slug} company and product journal # /benchmarks, /benchmarks/{section} proof dashboards # /partners, /pricing, /brand resources # /documentation, /documentation/ product docs + REST API and SDK reference # /login, /register auth entry points (thin, noindexed) # /legal/{terms,privacy,security,usage-policy,subprocessors} # # Authenticated app routes are JWT-gated, render no public content, and # would burn crawl budget for zero SERP value. They're explicitly # Disallowed below for every bot. # # AI crawler stance: we WELCOME ingestion by reputable AI training and # retrieval bots so Melaya surfaces in ChatGPT, Claude, Perplexity, # Gemini, SearchGPT, Apple Intelligence, Bing Copilot, Brave Leo, # DuckAssist, and CCBot-fed open-source LLMs. Every major AI bot has # an explicit allow block below the default rule. # ── Default rule (every crawler not listed below) ───────────────────────── User-agent: * # Public, indexable Allow: / Allow: /home # Products (all six, plus the demoted Melaya Labs trading pages) Allow: /product/ Allow: /product/agentic-framework Allow: /product/assistant Allow: /product/agentic-device-control Allow: /product/agentic-browser-control Allow: /product/mcp Allow: /product/marketing Allow: /product/agentic-trading-crew Allow: /product/trading-engine # Topic landing pages Allow: /enterprise-ai-agents Allow: /browser-ai-agents Allow: /mobile-ai-agents # Use cases, comparisons, guides Allow: /use-cases Allow: /use-cases/ Allow: /vs Allow: /vs/ Allow: /guides Allow: /guides/ # Content, proof, and resources Allow: /blog Allow: /blog/ Allow: /benchmarks Allow: /benchmarks/ Allow: /partners Allow: /partners/ Allow: /pricing Allow: /brand Allow: /documentation Allow: /documentation/ Allow: /catalog/ # Auth entry points — thin and noindexed at the page + X-Robots-Tag level, # but kept crawlable so the noindex is actually seen. Allow: /login Allow: /register # Legal Allow: /legal/terms Allow: /legal/privacy Allow: /legal/security Allow: /legal/usage-policy Allow: /legal/subprocessors # Assets + crawler files Allow: /og-image.png Allow: /og-socials.png Allow: /melaya_mark_dark_1024.png Allow: /favicon.ico Allow: /manifest.json Allow: /llms.txt Allow: /llms-fr.txt Allow: /llms-zh.txt Allow: /llms-es.txt Allow: /llms-pt.txt Allow: /llms-ru.txt Allow: /llms-tl.txt Allow: /llms-hi.txt Allow: /sitemap.xml Allow: /sitemap-index.xml Allow: /.well-known/ # Authenticated app routes — never index Disallow: /overview Disallow: /projects Disallow: /projects/ Disallow: /tracing Disallow: /tracing/ Disallow: /eval Disallow: /eval/ Disallow: /assistant Disallow: /assistant/ Disallow: /friday Disallow: /friday/ Disallow: /builder Disallow: /builder/ Disallow: /tools Disallow: /tools/ Disallow: /subagents Disallow: /subagents/ Disallow: /connectors Disallow: /connectors/ Disallow: /admin Disallow: /admin/ Disallow: /trading Disallow: /trading/ Disallow: /predictions Disallow: /predictions/ # Token-bearing URLs — keep them out of the index Disallow: /verify-email/ Disallow: /reset-password/ Disallow: /invite/ # Server endpoints — never crawl Disallow: /api/ # Static asset noise — bots don't need it Disallow: /src/ # ── AI training & retrieval crawlers (explicit allow) ───────────────────── # OpenAI (ChatGPT, SearchGPT, training corpus, GPT browsing) User-agent: GPTBot Allow: / Disallow: /api/ Disallow: /overview Disallow: /projects/ Disallow: /admin Disallow: /tracing/ Disallow: /verify-email/ Disallow: /reset-password/ Disallow: /invite/ User-agent: OAI-SearchBot Allow: / Disallow: /api/ Disallow: /overview Disallow: /admin Disallow: /verify-email/ Disallow: /reset-password/ Disallow: /invite/ User-agent: ChatGPT-User Allow: / Disallow: /api/ Disallow: /admin Disallow: /verify-email/ Disallow: /reset-password/ Disallow: /invite/ # Anthropic (Claude crawlers — both training and browsing) User-agent: ClaudeBot Allow: / Disallow: /api/ Disallow: /overview Disallow: /projects/ Disallow: /admin Disallow: /tracing/ Disallow: /verify-email/ Disallow: /reset-password/ Disallow: /invite/ User-agent: Claude-Web Allow: / Disallow: /api/ Disallow: /admin User-agent: claude-web Allow: / Disallow: /api/ Disallow: /admin User-agent: anthropic-ai Allow: / Disallow: /api/ Disallow: /admin # Google AI training (separate from Googlebot which already covers SERP) User-agent: Google-Extended Allow: / Disallow: /api/ Disallow: /overview Disallow: /admin # Google search SERP (explicit allow for completeness) User-agent: Googlebot Allow: / Disallow: /api/ Disallow: /overview Disallow: /admin # Perplexity User-agent: PerplexityBot Allow: / Disallow: /api/ Disallow: /admin User-agent: Perplexity-User Allow: / Disallow: /api/ # Microsoft / Bing AI + Copilot User-agent: bingbot Allow: / Disallow: /api/ Disallow: /overview Disallow: /admin User-agent: msnbot Allow: / Disallow: /api/ Disallow: /admin # Common Crawl (feeds many open-source LLMs) User-agent: CCBot Allow: / Disallow: /api/ Disallow: /admin # You.com User-agent: YouBot Allow: / Disallow: /api/ Disallow: /admin # Apple Intelligence (Siri + on-device AI) User-agent: Applebot Allow: / Disallow: /api/ User-agent: Applebot-Extended Allow: / Disallow: /api/ Disallow: /admin # Cohere User-agent: cohere-ai Allow: / Disallow: /api/ Disallow: /admin User-agent: cohere-training-data-crawler Allow: / Disallow: /api/ Disallow: /admin # Meta AI (Llama training) User-agent: FacebookBot Allow: / Disallow: /api/ Disallow: /admin User-agent: meta-externalagent Allow: / Disallow: /api/ Disallow: /admin # DuckAssist / DuckDuckGo User-agent: DuckAssistBot Allow: / Disallow: /api/ User-agent: DuckDuckBot Allow: / Disallow: /api/ # Brave Search / Brave Leo User-agent: BraveSearchBot Allow: / Disallow: /api/ Disallow: /admin User-agent: Brave-Search-Index Allow: / Disallow: /api/ # Mistral User-agent: MistralAI-User Allow: / Disallow: /api/ # Phind User-agent: PhindBot Allow: / Disallow: /api/ # Yandex (long-tail in Eastern Europe / CIS markets) User-agent: YandexBot Allow: / Disallow: /api/ Disallow: /admin # Naver (Korean market) User-agent: Yeti Allow: / Disallow: /api/ # Baidu (Chinese market) User-agent: Baiduspider Allow: / Disallow: /api/ # Discord link preview bot User-agent: Discordbot Allow: / Disallow: /api/ # Slack link preview bot User-agent: Slackbot Allow: / # LinkedIn link preview bot User-agent: LinkedInBot Allow: / # WhatsApp link preview bot User-agent: WhatsApp Allow: / # Telegram link preview bot User-agent: TelegramBot Allow: / # ── Sitemap pointers (parsed by every conforming crawler) ──────────────── # sitemap-index.xml is the authoritative entry point: it lists the eight # per-language sitemaps below. The legacy flat /sitemap.xml is deprecated and # 301-redirects to the index at the edge, so it is no longer advertised here. Sitemap: https://melaya.org/sitemap-index.xml Sitemap: https://melaya.org/sitemap-en.xml Sitemap: https://melaya.org/sitemap-fr.xml Sitemap: https://melaya.org/sitemap-zh.xml Sitemap: https://melaya.org/sitemap-es.xml Sitemap: https://melaya.org/sitemap-pt.xml Sitemap: https://melaya.org/sitemap-ru.xml Sitemap: https://melaya.org/sitemap-tl.xml Sitemap: https://melaya.org/sitemap-hi.xml # ── llms.txt pointer (proposed convention for AI ingestion) ─────────────── # Some AI crawlers honour a custom `Llm-Content` directive; sketching it # here as a forward-compatible hint (no harm if the parser ignores it). Host: melaya.org