Train the Conversation AI Bot with Web URLs in Sparkforge

This article covers training the Conversation AI bot with web URLs and what to watch for.

Steps to train with web URLs

1. Open Bot Training.
2. Enter a full URL (including https://) and pick a crawl mode (see below).
3. Wait for the URLs to fetch and crawl.
4. Select the URLs you want and click Train Bot.
5. Each trained URL appears in the Uploaded Links table with a status. Wait until all URLs finish before using the bot.

Limit: 4,000 web URLs per knowledge base.

Crawl modes

Exact URL

Recommended for precise training. The bot crawls only the URL you provide.

1. Pick Exact URL.
2. Enter the URL and click Get Data.
3. The page is crawled and added to the Uploaded Links table.

All URLs in this Domain

Crawls every page on the domain and lets you choose which to train on.

1. Pick All URLs in this domain.
2. Enter the domain and click Get Data.
3. Wait for the page list to load.
4. Select the relevant URLs and click Train Bot.

You'll see two lists during selection:

• New Pages — not yet trained. Selecting adds them to Uploaded Links once training finishes.
• Existing Pages — already trained. Selecting refreshes them.

All URLs with this Path

Same flow as Domain mode, but limits results to URLs containing the specified path.

Uploaded Links table

Every trained URL shows up here. You can refresh a URL (re-crawl for updated content) or delete it (removes it from the bot's knowledge).

Status values:

• Getting Data — crawl in progress.
• Trained — successful. Last refresh timestamp is shown.
• Failed — refresh to retry, or delete.

URLs aren't added instantly. Wait until all entries appear before testing the bot.

Keep training data tight. Fewer, more relevant URLs produce better answers than thousands of stale pages. Review and prune the table on a regular cadence.