Index a whole documentation site

Every page of a docs site as clean text, without writing a crawler: /v1/map finds the URLs, /v1/batch reads them, and you store what comes back.

curl -X POST https://api.buildonto.dev/v1/map \
  -H "Authorization: Bearer $ONTO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docs.anthropic.com", "limit": 500}'

It reads /sitemap.xml (and up to three sitemaps it points to). With no sitemap, it lists the links on the start page instead — source says which. limit defaults to 100 and tops out at 1,000. One credit per call.

const KEY = process.env.ONTO_API_KEY!;
const BATCH = 50; // /v1/batch reads at most 50 — the rest are dropped, silently
const post = (path: string, body: unknown) =>
  fetch(`https://api.buildonto.dev${path}`, {
    method: "POST",
    headers: { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" },
    body: JSON.stringify(body),
  }).then((r) => r.json());

// 1. discover
const { urls, source } = await post("/v1/map", {
  url: "https://docs.anthropic.com",
  limit: 500,
});
console.log(`${urls.length} urls from ${source}`); // "sitemap" or "links"

// 2. read, fifty at a time
const pages: { url: string; title: string; markdown: string; truncated: boolean }[] = [];

for (let i = 0; i < urls.length; i += BATCH) {
  const chunk = urls.slice(i, i + BATCH);
  const res = await post("/v1/batch", { urls: chunk, mode: "read" });

  for (const r of res.results) {
    if (!r.ok) {
      // a dead link in a sitemap is normal — keep going (and it's refunded)
      console.warn("skipped", r.url, r.error.code);
      continue;
    }
    pages.push({
      url: r.url,
      title: r.title,
      markdown: r.markdown,
      truncated: r.markdown.includes("[truncated by /v1/batch"),
    });
  }

  console.log(`${pages.length}/${urls.length} read`);
}

Each batch costs a credit per URL, fetches eight at a time, and holds one of your concurrent slots while it runs. Leave out mode and you get read-and-score, which adds each page's score for the same credit.

A page over 12,000 characters comes back ending in …[truncated by /v1/batch — call /v1/read on this URL for the full page]. Most docs pages never hit it; long API references hit it constantly.

// re-read only the pages that got cut, at full length
const cut = pages.filter((p) => p.truncated);
console.log(`${cut.length} pages were over the batch cap`);

for (const page of cut) {
  const full = await post("/v1/read", { url: page.url });
  page.markdown = full.markdown;
  page.truncated = false;
}

/v1/read has no length cap (pages up to 10 MB). One credit per page re-read.

Keep the source URL, so you can cite it, and the date, so you can re-run and diff.

await db.insert(
  pages.map((p) => ({
    url: p.url,
    title: p.title,
    content: p.markdown,
    indexed_at: new Date().toISOString(),
  })),
);

A re-run is charged again, cached or not — and the batch cache only matches the exact same set of URLs. Re-index on a schedule you choose, not on every deploy of your app.

Map found one URL

No sitemap, and no links on the start page. Map doesn't crawl further — pass a URL list you already have straight to step 2.

Some URLs never came back

You sent more than 50 in one call. The extras are dropped without an error — chunk them.

Everything is ROBOTS_BLOCKED

The site disallows GPTBot or *. There's no flag to override it; only a site you've verified in Serve is exempt.

Pages end in “truncated by /v1/batch”

The 12,000-character cap. That's what step 3 is for.

CONCURRENT_LIMIT

More batches or reads in flight than your plan's slots — often another process or an MCP server on the same key. Wait retry_after (one second) and try again.