Discover the URLs on a website

8 providers do this through one terra key: AnyAPI, Context.dev, Crawl4AI, Firecrawl, Olostep, Search1API, Spider, Tavily.

AnyAPI Map an entire website into a clean list of its URLs (with titles and descriptions) in a single call.$0.001/success · anyapi.web.map
Context.dev Discover bounded lists of URLs on a domain without fetching page bodies.$0.00253/success · branddev.web.map
Crawl4AI A site's URLs from the sources that already know them: its sitemaps, the links on its home page and the newest Common Crawl indexes (mode full adds the Wayback Machine). One call, 1 to 3 seconds for most sites. Each row: url, sources, last_seen, depth.$0.00015/success · crawl4ai.web.map
Firecrawl Discover URLs on a website without scraping their content.$0.005/success · firecrawl.web.map
Olostep Map a site and return at most 1,000 URLs ranked for a search query.$0.002/call · olostep.web.map.search
Search1API List URLs from a site's sitemap or discovered links.$0.001/success · search1api.web.sitemap
Spider Find URLs on a website without returning full page bodies.own account · spidercloud.web.links
Tavily Discover URLs from a website with depth, breadth, path and domain controls.$0.0008-$0.0016/success · tavily.web.map

Loading terra…