# Index a whole documentation site

> Every page of a docs site as clean text without writing a crawler: map, batch in fifties, re-read what got cut.

Source: https://docs.buildonto.dev/guides/knowledge-base
Section: Guides

---

Index a whole site\[01\]

## Index a whole documentation site

Every page of a docs site as clean text, without writing a crawler: `/v1/map` finds the URLs, `/v1/batch` reads them, and you store what comes back.

### The whole pipeline

\[02\]

api.buildonto.dev · map → batch → read

One docs URL

1.  01Find the pages
    
    /v1/map reads `sitemap.xml` — or, without one, the links on the start page. Same host only, up to 1,000 URLs.
    
2.  02Split into fifties
    
    /v1/batch takes 50 URLs per call. Anything past 50 is dropped without an error.
    
3.  03Read each chunk
    
    Inside a batch, eight pages are fetched at a time, round by round. Failed URLs come back `ok: false` and are refunded.
    
4.  04Re-read what got cut
    
    Batch cuts each page at 12,000 characters and says so at the end. `/v1/read` has no length cap.
    
5.  05Store it
    
    Keep the URL to cite and the date to re-run against.
    

{ url, title, markdown } for every page, ready to chunk and embed

**Two caps shape the loop.** `/v1/batch` takes at most **50 URLs** per call and drops the rest without saying so, and it cuts each page at **12,000 characters**. Steps 2 and 3 handle both.

### 1\. Find the pages

\[03\]


```
curl -X POST https://api.buildonto.dev/v1/map \
  -H "Authorization: Bearer $ONTO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docs.anthropic.com", "limit": 500}'
```

It reads `/sitemap.xml` (and up to three sitemaps it points to). With no sitemap, it lists the links on the start page instead — `source` says which. `limit` defaults to 100 and tops out at 1,000. One credit per call.

### 2\. Read them in batches

\[04\]

typescriptindex-site.tsCopy

```
const KEY = process.env.ONTO_API_KEY!;
const BATCH = 50; // /v1/batch reads at most 50 — the rest are dropped, silently
const post = (path: string, body: unknown) =>
  fetch(`https://api.buildonto.dev${path}`, {
    method: "POST",
    headers: { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" },
    body: JSON.stringify(body),
  }).then((r) => r.json());

// 1. discover
const { urls, source } = await post("/v1/map", {
  url: "https://docs.anthropic.com",
  limit: 500,
});
console.log(`${urls.length} urls from ${source}`); // "sitemap" or "links"

// 2. read, fifty at a time
const pages: { url: string; title: string; markdown: string; truncated: boolean }[] = [];

for (let i = 0; i < urls.length; i += BATCH) {
  const chunk = urls.slice(i, i + BATCH);
  const res = await post("/v1/batch", { urls: chunk, mode: "read" });

  for (const r of res.results) {
    if (!r.ok) {
      // a dead link in a sitemap is normal — keep going (and it's refunded)
      console.warn("skipped", r.url, r.error.code);
      continue;
    }
    pages.push({
      url: r.url,
      title: r.title,
      markdown: r.markdown,
      truncated: r.markdown.includes("[truncated by /v1/batch"),
    });
  }

  console.log(`${pages.length}/${urls.length} read`);
}
```

Each batch costs a credit per URL, fetches eight at a time, and holds one of your concurrent slots while it runs. Leave out `mode` and you get `read-and-score`, which adds each page's score for the same credit.

**Running chunks in parallel?** Each batch holds one slot, so you can run as many at once as your plan allows — 2 on Free, 5 on Starter, 20 on Growth, 50 on Scale. One more than that, and you get `CONCURRENT_LIMIT`.

### 3\. Re-read what got cut

\[05\]

A page over 12,000 characters comes back ending in `…[truncated by /v1/batch — call /v1/read on this URL for the full page]`. Most docs pages never hit it; long API references hit it constantly.


```
// re-read only the pages that got cut, at full length
const cut = pages.filter((p) => p.truncated);
console.log(`${cut.length} pages were over the batch cap`);

for (const page of cut) {
  const full = await post("/v1/read", { url: page.url });
  page.markdown = full.markdown;
  page.truncated = false;
}
```

`/v1/read` has no length cap (pages up to 10 MB). One credit per page re-read.

### 4\. Store it

\[06\]

Keep the source URL, so you can cite it, and the date, so you can re-run and diff.


```
await db.insert(
  pages.map((p) => ({
    url: p.url,
    title: p.title,
    content: p.markdown,
    indexed_at: new Date().toISOString(),
  })),
);
```

A re-run is charged again, cached or not — and the batch cache only matches the exact same set of URLs. Re-index on a schedule you choose, not on every deploy of your app.

### What it costs

\[07\]

Pages on the site **240**Longer than 12,000 characters **5%** _your guess_

1.  Map1 call1
2.  Batch5 calls of up to 50240
3.  Re-read cut pages12 × /v1/read12
4.  Creditsfits in Free (1,000 a month)253

Failed reads are refunded. A re-run costs the same again — cached or not.

### Where it goes wrong

\[08\]

Map found one URL

No sitemap, and no links on the start page. Map doesn't crawl further — pass a URL list you already have straight to step 2.

Some URLs never came back

You sent more than 50 in one call. The extras are dropped without an error — chunk them.

Everything is ROBOTS\_BLOCKED

The site disallows `GPTBot` or `*`. There's no flag to override it; only a site you've verified in Serve is exempt.

Pages end in “truncated by /v1/batch”

The 12,000-character cap. That's what step 3 is for.

CONCURRENT\_LIMIT

More batches or reads in flight than your plan's slots — often another process or an MCP server on the same key. Wait `retry_after` (one second) and try again.

Copy as Markdown[](/guides/knowledge-base.md "Open the raw Markdown")

---
## Structured Data (JSON-LD)
```json
{
  "@context": "https://schema.org",
  "@type": "WebSite",
  "name": "Onto Docs",
  "url": "https://docs.buildonto.dev",
  "description": "How to use Onto: serve AI agents Markdown from your Next.js site, call the Read API, connect over MCP, and read the AIO score.",
  "inLanguage": "en",
  "publisher": {
    "@type": "Organization",
    "name": "Onto",
    "url": "https://buildonto.dev"
  }
}
```