# robots.txt — ortexo.com # # Machine-readable summary of what this site covers, for LLMs and AI crawlers: # https://ortexo.com/llms.txt # # The policy in one line: search engines and AI *answer* traffic are welcome, # bulk AI *training* crawls and commercial SEO scrapers are not. # # That distinction is the whole point and is easy to get wrong, because the same # company often runs both. GPTBot crawls to train; ChatGPT-User fetches one page # because somebody just asked about it. ClaudeBot trains; Claude-User answers. # Google-Extended feeds Gemini's training; Googlebot feeds Search. Blocking the # first of each pair costs nothing in visibility. Blocking the second would # remove this site from AI answers, which is not the intent. # # TEXT AND DATA MINING IS RESERVED. Poly Tool Design SRL reserves, in machine-readable # form, the rights over text and data mining of this site's contents (Directive (EU) # 2019/790 art. 4(3); Law 8/1996 art. 36^2): the `tdm-reservation: 1` response header, # /.well-known/tdmrep.json, and the terms of use at https://ortexo.com/terms-of-use, # which also forbid bots, scrapers and bulk extraction without prior written consent. # # Keep this in step with CRAWLER_UA in src/lib/cidr.js. That list decides who is # exempt from the per-IP read budget, and the two disagreeing is how the site # ended up promising `Allow: /` here while returning 429 at the edge. # --- Standard crawlers --- # /api/ is blocked, except two paths of the data proxy. Every image inside a legal document is # served through the first, so a blanket block keeps the whole corpus out of image search. The # tables of an act (rates, thresholds, annexes) are loaded by the page's script from the second, # so a blanket block meant Google rendered every act without its tables (OrtexoWeb-axw0.8). The # longer Allow wins over `Disallow: /api/` for Googlebot and Bingbot. The edge gate still asks # every other reader for the reader cookie and the budget; only verified crawlers skip them. User-agent: * Allow: /api/proxy/nodes/image Allow: /api/proxy/nodes/table-rows Allow: / Disallow: /admin/ Disallow: /api/ # --- Google --- User-agent: Googlebot Allow: /api/proxy/nodes/image Allow: /api/proxy/nodes/table-rows Allow: / Disallow: /admin/ Disallow: /api/ # --- Bing --- User-agent: Bingbot Allow: /api/proxy/nodes/image Allow: /api/proxy/nodes/table-rows Allow: / Disallow: /admin/ Disallow: /api/ # --- AI answer traffic: allowed -------------------------------------------- # These fetch a page because a user asked a question about it, or index it so it # can be cited in an answer. This is the channel that puts Ortexo in front of # people asking legal questions, and it is low volume by nature. User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # Apple's search crawler (Siri, Spotlight). Its training sibling is refused below. User-agent: Applebot Allow: / # --- AI training crawls: refused ------------------------------------------- # Measured over seven days before this rule: ClaudeBot alone took 90,100 requests # and 2.42 GB — more than double Googlebot — and 99.4% of it was redirects, so # almost none of that transfer even reached a document. # # Google-Extended is Gemini training ONLY. Refusing it does not affect Google # Search ranking or indexing in any way; Googlebot above is untouched. User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: meta-externalagent Disallow: / User-agent: FacebookBot Disallow: / User-agent: cohere-ai Disallow: / User-agent: Diffbot Disallow: / User-agent: omgili Disallow: / User-agent: Timpibot Disallow: / User-agent: Amazonbot Disallow: / # Amazon's second agent. Amzn-SearchBot appeared in the logs at 735 requests # within hours of Amazonbot being refused and starting to wind down — the same # company continuing the same crawl under a new name, and unlisted, so # `User-agent: *` was letting it straight back in. Refusing one agent per # incident is a losing game, so the known Amazon strings are all named here. User-agent: Amzn-SearchBot Disallow: / User-agent: Amazon-Kendra Disallow: / User-agent: AmazonAdBot Disallow: / User-agent: meta-webindexer Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: AI2Bot Disallow: / User-agent: Webzio-Extended Disallow: / User-agent: PanguBot Disallow: / User-agent: Kangaroo Bot Disallow: / User-agent: SemrushBot-OCOB Disallow: / User-agent: VelenPublicWebCrawler Disallow: / User-agent: Scrapy Disallow: / # --- Commercial SEO scrapers: refused --------------------------------------- # Backlink-index crawlers. They return nothing to this site and cost real # transfer: AhrefsBot ran ~30 IPs, ~25,000 requests and ~1.2 GB in seven days, # ~99% of it redirects. The others are the same business, so they get the same # answer. User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: dataforseo-bot Disallow: / User-agent: BLEXBot Disallow: / User-agent: Barkrowler Disallow: / User-agent: panscient.com Disallow: / User-agent: panscient Disallow: / # --- Deliberately NOT refused --- # facebookexternalhit link previews when someone shares a page in Messenger or # WhatsApp. Blocking it breaks the preview card, not a crawl. # YandexBot a real search engine with genuine share in this region. # Applebot Siri and Spotlight discovery. Its training sibling, # Applebot-Extended, is refused above. # Claude-SearchBot, the AI citation channel. Measured: after the URL migration # OAI-SearchBot settled these fell 61% and 38% respectively, so the cost is # far lower than it first appeared, and being the cited source # in an AI answer is high-intent discovery for a legal corpus. # # --- Sitemaps --- Sitemap: https://ortexo.com/sitemap.xml