Skip to content
Skryp

Crawl a site or list its URLs

Map lists a site's URLs from its sitemaps and links in seconds. Crawl reads pages in the background with path filters and depth limits; jobs survive restarts and can be cancelled or resumed. Each page is stored and readable in sections.

Try it: find all pages on a website

  • Sitemap-first discovery, newest sitemaps first.
  • Include and exclude path patterns, depth and page limits.
  • Cancel, resume, and a webhook when the crawl finishes.

Limits

  • Up to 5,000 pages per crawl.
  • Polite per-host concurrency.

Price

Map: 1 credit. Crawl: each page costs what its route costs. Full rate card

Start freeRead the quickstart
Crawl the docs · REST
curl -X POST https://api.skryp.dev/v1/crawl \
  -H "Authorization: Bearer $SKRYP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://docs.example.com",
  "limit": 50,
  "include_paths": [
    "^/docs/"
  ]
}'
Crawl the docs · MCP tool call
{
  "tool": "crawl",
  "arguments": {
    "url": "https://docs.example.com",
    "limit": 50,
    "include_paths": [
      "^/docs/"
    ]
  }
}

Related