Submit Feedback

Data Sources

Add an external website as a data source so agents can answer from that site’s content. Without it, an agent only knows what’s in its model and the content you give it directly. AI Hub crawls the linked website, stores the extracted text, and makes it available to any agent you attach the data source to. Chatbots use the data source through the agents they call. This grounding of answers in retrieved content is called Retrieval-Augmented Generation (RAG).

For example, point a data source at your public product documentation site so a support agent can answer questions from it.

A data source has a name, a single external website URL, and a description. File-based data sources aren’t available.

Creating a Data Source

Create a data source for the website you want agents to draw on:

  1. Click Agent Builder in the AI Hub left navigation, then select the Data Sources tab.

  2. Click New Data Source.

  3. Enter a name, the website URL in the External Website field, and a description.

  4. Click Save.

The Add URL form has fields for a name, the external website URL, and a required description.

The data source appears on the Data Sources tab. Use its Actions Actions menu to view, edit, or delete it later.

Crawling a Website

Creating a data source doesn’t crawl the site. To extract the content, open the data source’s Actions Actions menu and click Sync Now.

AI Hub crawls the site within fixed limits:

  • Depth: three levels of links from the URL
  • Scope: the same domain only, with no cross-domain crawling
  • Page limit: 500 pages maximum

Each crawled page becomes a resource within the data source. Crawling is manual, with no automatic re-crawl, so click Sync Now again whenever you want to refresh the content. Configurable crawl depth, URL patterns, and scheduling aren’t available.

Attaching a Data Source to an Agent

When you build an agent, the agent form’s Data Sources section has an Assigned Sources field where you select one or more data sources to extend its context. At runtime, the agent queries the attached data sources with semantic search, matching on meaning rather than exact keywords, to find the most relevant content. Chatbots draw on data sources through the agents they call.

For a walkthrough of building an agent, see Building a Custom Agent.