Crawl a website and extract its pages

9 providers do this through one terra key: AnyAPI, Context.dev, Crawl4AI, Firecrawl, Olostep, ScrapeGraphAI, Search1API, Spider, Tavily.

AnyAPI Crawl a website and get clean text content from up to 10 pages in one normalized response, ideal for feeding sites into LLMs and search indexes.$0.0045/success · anyapi.web.crawl
Context.dev Return Markdown from a bounded crawl of linked pages.$0.00253-$0.0253/success · branddev.web.crawl
Crawl4AI Crawls one site level by level from a start URL, inside the site, through include and exclude filters, cut at max_pages; runs only as a job. Poll it with crawl4ai.web.scrape.job.status and read the pages (url, level, title, status, links, markdown or html) with crawl4ai.web.scrape.job.results.$0.0751/success · crawl4ai.web.crawl
Firecrawl Start a bounded crawl and pay for pages processed.$0.005-$0.05/success · firecrawl.web.crawl
Olostep Start a bounded asynchronous website crawl and pay only for completed pages.$0.002-$0.2/success · olostep.web.crawl
ScrapeGraphAI Start a bounded one-page Markdown crawl and return its job id.$0.012/success · scrapegraphai.web.crawl
Search1API Start a site crawl and later retrieve its ZIP URL from task status.$0.02/success · search1api.web.deepcrawl
Spider Follow links from a starting URL and return a bounded set of pages.own account · spidercloud.web.crawl
Tavily Map a website and extract clean content from the discovered pages in one request.$0.0024-$0.0048/success · tavily.web.crawl

Loading terra…