Enhanced Web Crawler for Conversation AI in Sparkforge

By Sparkforge Support Team ·Last reviewed May 3, 2026

The Enhanced Web Crawler ingests dynamic site content — tabs, accordions, lazy-loaded sections — so Sparkforge Conversation AI bots train on 30–50% more on-page text than a standard crawl.

Static HTML crawlers miss most of what modern websites actually display. The Enhanced Web Crawler in Sparkforge Conversation AI mimics real visitor behavior — opening accordions, clicking tabs, scrolling lazy-loaded sections — so your bot trains on the full page, not just the markup.

What the Enhanced Web Crawler does

It is the upgraded ingestion engine inside Bot Training. It runs alongside the existing Exact URL, Domain, and Path crawlers, but adds interaction simulation so dynamic content shows up in the training set.

Deeper text capture: 30–50% more content from React, Vue, Angular, and Gutenberg sites.
Hidden content awareness: reads accordions, tabs, modals, infinite scroll.
Parallel parsing: 12+ content-detection strategies run at once.
Safe interaction: avoids form submits, filter changes, and cart actions.
Crawl metrics: processing time, interactions, content length, memory.

Link discovery improvements

Recursive sitemap crawling for nested sitemap structures.
Compressed sitemap support (.xml.gz, .gzip).
Navigation guard keeps the crawler inside the intended scope.
HTML parsing plus JavaScript evaluation plus interaction-based discovery.
Deduplication that preserves descriptive anchor text.

How to run an Enhanced Web Crawl

In your sub-account, open AI Agents.
Click the Knowledge Base tab.
Create a new Knowledge Base or edit an existing one.
Click + Add Source, then Web Crawler.
Pick a domain type (Exact URL, Path, or Full Domain) and enter the URL.
Enable Enhanced Crawl, then start the job.

Domain type options

Exact URL: only the page you specify.
All URLs with the path: every page under a path like /marketing.
Full domain: the entire site within sitemap and link-discovery limits.

What it works on

Static HTML, WordPress, React SPAs, Vue, and Angular apps. Parallel extraction shortens crawl time on large sites, and the metrics view tells you where time was spent.

Frequently Asked Questions

How much more content does the Enhanced Crawler capture?

Roughly 30–50% more text on modern single-page apps, because it expands accordions, tabs, and lazy-loaded sections that a static fetch would skip.

Will the crawler submit forms or click buy buttons?

No. The interaction engine is restricted to safe actions — opening accordions, switching tabs, scrolling — and avoids form submissions, filter changes, and cart actions.

Does it support compressed sitemaps?

Yes. It reads .xml.gz and .gzip sitemaps and recursively follows nested sitemap indexes.

Can I scope a crawl to a single section of my site?

Yes. Use the All URLs with the Path option and pass a path like /help so the crawler stays inside that section.

Where do I see crawl performance?

Each crawl logs processing time, content length, interaction count, and memory usage so you can spot slow pages or thin content.