Find all pages on a website

Signed in, the tool maps your site live on your workspace (1 credit for up to 5,000 addresses) and lets you download the list as CSV. Without an account it shows the recorded example and takes you to sign-up with your address kept.

How the pages are found

1. Sitemaps first. Skryp reads the sitemaps your site lists in its robots.txt, or /sitemap.xml when it lists none, and follows sitemap indexes to the sitemaps inside them, newest first.
2. Then the links on the page you entered. Anything linked from it that the sitemaps missed is added and marked as found on the page.
3. On the same host. Links to other sites and to subdomains are left out, so example.com does not pull in blog.example.com unless you ask for subdomains through the API.

The count for each method shows how complete your sitemap is: in the recorded example, 1,016 of Django's 1,018 addresses came from its sitemap.

What this does not find

• Pages nobody links to and no sitemap lists. A crawl that follows links several levels deep finds more: see Crawl and map.
• Pages behind search or a sign-in. Results pages and account areas are not listed anywhere a crawler can see.
• Whether each page works. The list is addresses; to check them, scrape them. Web crawling vs web scraping shows the two steps together.

The same call from your code or your agent

# List a site's pages from its sitemaps and its links with the Skryp API, and print what the tool shows.
# pip install httpx    export SKRYP_API_KEY=...
import json
import os
from collections import Counter

import httpx

API = os.environ.get("SKRYP_API_URL", "https://api.skryp.dev")
site = httpx.post(
    f"{API}/v1/map",
    headers={"Authorization": f"Bearer {os.environ['SKRYP_API_KEY']}"},
    json={"url": "https://www.djangoproject.com/", "limit": 5000},
    timeout=120,
).json()

print(json.dumps({
    "url": site["url"], "count": site["count"], "sitemaps_read": site["sitemaps_read"],
    "by_source": Counter(link["source"] for link in site["links"]),
    "credits": site["receipt"]["credits"],
    "links": site["links"][:60],
}, indent=1))

Over MCP it is the map tool with the site's url. Add search to keep only addresses that contain a word, such as "blog".