Knowledge sources

A knowledge source is one input the AI agent is allowed to answer from. Sources live in Knowledge base → Sources; the Test retrieval tab next to it shows the excerpts an agent would receive for a given question.

Nothing is indexed implicitly. Each source is created explicitly, indexed on its own schedule, and can be selected or deselected per agent in AI answers and agents.

Source types

  • Knowledge base article — an existing article of this bot. Published articles become public sources, internal articles become agent-only sources, and drafts produce no searchable content at all.
  • Website — pages of your own site, fetched by an allowlisted crawler.
  • File — a PDF, Word, Excel, PowerPoint, text, Markdown, or HTML upload.
  • Instruction — a short piece of policy text you write yourself, for example a refund rule the agent must always follow.
  • Fact — a single stored fact referenced by its identifier.

The source name is limited to 120 characters, and the whole source configuration — including instruction text — to 64 KB.

Visibility and priority

Every source has one visibility:

  • AI agent — only the agent can retrieve it.
  • Operators — reserved for operator-facing retrieval.
  • Public — may also appear in public surfaces such as the help center.

Files, instructions, and facts can never be public; only knowledge base articles reach public visibility, and they do it through their own published state. Visibility is part of the database query, not a filter applied afterwards, so an agent restricted to public content cannot receive an internal excerpt.

Priority ranges from −100 to 100. It is a tie-break: when two excerpts rank equally, the higher-priority source wins. It never overrides a clearly better text match.

Website sources

A website source needs at least one allowed origin (a bare https://host address, up to 20 of them) and at least one start URL inside those origins, up to 100 URLs. HTTP addresses are rejected.

Crawling stays inside those boundaries:

  • The crawler identifies itself as MyBotKnowledge/1.0 and reads robots.txt for every origin before fetching. Disallow prefixes are skipped and Crawl-delay is honoured up to 10 seconds.
  • Only <a href> links that resolve back into the allowed origins are followed. Fragments are dropped and the utm_*, gclid, and fbclid parameters are removed before a URL is queued, so the same page is not indexed twice.
  • Each request re-resolves the host and refuses loopback, private, link-local, multicast, and cloud metadata addresses. Redirects are re-validated against the same rules, and at most three are followed.
  • A request times out after 15 seconds, the response body is capped at 8 MB, and only HTML, XHTML, plain text, and Markdown responses are accepted.
  • A single source indexes at most 100 pages.

The crawler never signs in, never submits a form, and never loads scripts, media, or embedded resources. Pages behind a login are therefore out of scope: publish the content or upload it as a file instead.

Refresh every (seconds) applies to website sources only. It must be at least 60 seconds and at most 366 days. Article, file, instruction, and fact sources are not re-fetched on a schedule; they are re-indexed when you change them or press Refresh.

File uploads

Uploads must already belong to this bot's media library and be at most 8 MB. The accepted types are PDF, DOCX, XLSX, PPTX, text/plain, text/markdown, text/html, and XHTML.

Extraction is deliberately narrow:

  • HTML keeps headings, lists, and link text; scripts, styles, navigation, headers, footers, asides, and templates are stripped.
  • PDFs are read as text only, up to 100 pages. Encrypted or malformed files are rejected instead of being partially parsed.
  • Office files are read from their document text parts only. Macros, embedded objects, and external relationships are ignored and never fetched. The archive must stay under 500 entries, a 100:1 compression ratio, 32 MB expanded, 200 000 XML nodes, and 500 000 characters of text.

Legacy binary .doc, .xls, and .ppt files are rejected with unsupported_type. Save them as DOCX, XLSX, or PPTX and upload again.

Indexing and statuses

A source moves through these states, shown on its card:

  • Waiting to index — created or changed, not processed yet.
  • Indexing — a worker holds the job.
  • Ready — the current content is searchable.
  • Needs refresh — the schedule is due or the content changed.
  • Failed — the last attempt ended with an error code.
  • Disabled — kept but excluded from every search.

Indexing is staged. A worker reads the source, splits the text into overlapping fragments (about 1 500 characters with 200 characters of overlap), requests embeddings in batches of 32, and only then replaces the source's documents and fragments in a single transaction. A reader always sees either the previous complete version or the new complete version, never a half-rebuilt source. If the content checksum has not changed, the rebuild is skipped.

Embeddings are produced through the bot's AI integration (Integrations), using its base URL and key, and default to the text-embedding-3-small model with 1 536 dimensions. Without a working AI integration a source cannot leave the indexing queue.

One source indexes at most 100 documents, and a single document may not produce more than 500 fragments. Jobs are leased for two minutes with a heartbeat, so an interrupted worker releases the source instead of blocking it. The source card shows the attempt count and the number of processed documents, processed fragments, and failed documents; Cancel indexing stops the current run.

What retrieval returns

A question is answered from two independent searches:

  1. a lexical search over the PostgreSQL text index, and
  2. a semantic search over the vector index (cosine distance).

The bot, the visibility set, the active-source condition, and the not-deleted-document condition are all part of both SQL queries, before ranking and before any limit. The two rankings are then fused with reciprocal rank fusion, source priority and recent indexing (within 30 days) break ties, and neighbouring fragments of the same document are collapsed.

The result is at most 12 excerpts of at most 800 characters each. A question is limited to 16 000 characters, and an agent may be restricted to at most 100 specific sources.

The agent stores the source, document, and fragment behind every excerpt it received, in rank order. An answer that cites anything else is rejected before it is sent, so a reply can never point to a draft, hidden, disabled, or deleted source.

Testing retrieval

Knowledge base → Test retrieval runs the same search without involving the model. For each hit it shows the source, the title, the excerpt, the match score, and the reason (Hybrid match). Hidden metadata is not returned, and the test is rate limited per account.

Use it before activating an agent: if the excerpts here are wrong, the answer will be wrong too.

When indexing fails

The last failure is stored as a stable code:

  • blocked_address — the URL, a redirect, or the resolved IP is outside the allowed origins or points at a private address.
  • robots_denied — robots.txt forbids that path.
  • unsupported_type — the content type or file format is not supported.
  • file_too_large — the download or upload exceeds the size limit.
  • expanded_too_large — the extracted text, page count, or archive contents exceed the limits above.
  • malformed_file — the file or response could not be parsed.
  • fetch_timeout — the site did not answer in time.
  • rate_limited — the embedding provider throttled the request; the job is retried.

Fix the cause and press Refresh. Errors never include the fetched text, so they are safe to share with support.

Deleting, disabling, and retention

Disable keeps the configuration but removes the source from every search; Enable puts it back in the queue. Delete removes the source together with its documents and fragments.

A source that is selected by an active agent cannot be deleted — the request fails with a conflict. Deselect it in the agent first, or pause the agent.

Isolation

Sources, documents, fragments, and index jobs are all keyed by bot. Every read path filters on the bot inside SQL, so one tenant's content is never a candidate for another tenant's search, even when the text and the embeddings are identical.

API and MCP

The REST API uses the knowledge:read and knowledge:write scopes:

  • GET/POST /api/bots/{botID}/knowledge/sources
  • GET/PATCH/DELETE /api/bots/{botID}/knowledge/sources/{sourceID}
  • POST /api/bots/{botID}/knowledge/sources/{sourceID}/upload
  • POST /api/bots/{botID}/knowledge/sources/{sourceID}/refresh
  • POST /api/bots/{botID}/knowledge/sources/{sourceID}/cancel
  • POST /api/bots/{botID}/knowledge/sources/{sourceID}/disable
  • POST /api/bots/{botID}/knowledge/sources/{sourceID}/enable
  • GET /api/bots/{botID}/knowledge/sources/{sourceID}/index-status
  • GET /api/bots/{botID}/knowledge/sources/{sourceID}/documents
  • POST /api/bots/{botID}/knowledge/retrieval-test

MCP exposes read-only tools: list_knowledge_sources, get_knowledge_source_status, and test_knowledge_retrieval. There is no MCP tool that changes a source or downloads raw hidden documents.

Related pages