ROBOTS_BLOCKED[01]

ROBOTS_BLOCKED

The site's robots.txt has Disallow: / for GPTBot or for every user agent.

Code · HTTP status · Retryable?
ROBOTS_BLOCKED[403][No]

What this means

[02]

Onto reads `/robots.txt` before it fetches the page. If a `User-agent: GPTBot` or `User-agent: *` group contains `Disallow: /`, the request stops with ROBOTS_BLOCKED. Only a whole-site `Disallow: /` counts; rules for specific paths are ignored. If robots.txt fails to load, Onto treats the site as allowed. Failed requests are refunded.

When you'll see it

[03]

HTTP 403. Body always includes code: "ROBOTS_BLOCKED". Branch on code, never on the human-readable message — wording can change without notice; the code is the stable contract.

Example response

[04]
example response[json]
{
  "status": "error",
  "code": "ROBOTS_BLOCKED",
  "message": "Target site's robots.txt disallows AI crawlers. We honor robots.txt — see https://buildonto.dev/about-bot."
}

How to handle

[05]

Honor the block. Tell your user the site has opted out of AI crawlers and link them to the source. If it's your own site, allowing GPTBot is not enough while `User-agent: *` still has `Disallow: /`. Verify the domain in Serve instead: Onto then lets the verified owner read it. That doesn't apply to /v1/score.

Suggested handling in a Node client:

if (data.code === 'ROBOTS_BLOCKED') {
  // The site explicitly opted out. Honor it.
  return null;
}
Code · Status · What it means
WAF_BLOCKED[HTTP 403]

The target site refused Onto's crawler (the origin returned 401 or 403).

URL_NOT_FOUND[HTTP 404]

The target URL returned 404, or its hostname doesn't resolve.

See the full error index for the complete catalog with the handling switch statement covering every code at once.