CVE-2026-46497

Crawlee is a web scraping and browser automation library. From version 1.0.0 to before version 1.7.0, Crawlee is vulnerable to SSRF via sitemap-derived URLs. This issue has been patched in version 1.7.0.

Crawlee is a web scraping and browser automation library. From version 1.0.0 to before version 1.7.0, Crawlee is vulnerable to SSRF via sitemap-derived URLs. This issue has been patched in version 1.7.0.

Overview

• Vulnerability type: Blind SSRF
• Affected components: src/crawlee/_utils/sitemap.py, src/crawlee/_utils/robots.py, src/crawlee/request_loaders/_sitemap_request_loader.py, and all built-in HTTP clients.
• Trigger: an attacker-controlled sitemap or robots.txt containing a URL that points to an internal host (layer 1) or uses a non-http scheme (layer 2).

Two-layer SSRF via sitemap-derived URLs:

1) Cross-host HTTP SSRF

Base case, affects every HTTP client.** Sitemap entries and robots.txt Sitemap: directives were accepted regardless of the host they pointed to. A sitemap on example.com could push http://internal.corp/admin into the crawler's queue, and the configured HTTP client would dispatch the request.

2) Non-HTTP scheme SSRF

Escalation, only CurlImpersonateHttpClient.** Nested-sitemap fetching dispatches the URL straight to the HTTP client, bypassing the Request construction step where Pydantic enforces http(s). Combined with the libcurl-backed CurlImpersonateHttpClient, this lets gopher://, file://, dict://, ftp://, etc., through.

Root cause

Crawlee already validates URL schemes through Pydantic's AnyHttpUrl (via validate_http_url in src/crawlee/_utils/urls.py) wherever a crawl target is materialised as a Request: the Request.url field is declared as Annotated[str, BeforeValidator(validate_http_url), Field(frozen=True)]. Anything that becomes a Request is therefore guaranteed to be http(s).

Two parts of the sitemap pipeline sidestepped this property in different ways:

1) Sitemap-derived URLs were enqueued without any host policy

SitemapRequestLoader took every <urlset><url><loc> entry, wrapped it in Request.from_url (which accepts any valid http(s) URL), and pushed the result into the request queue. RobotsTxtFile.get_sitemaps() returned every Sitemap: directive verbatim. Neither imposed any host check against the parent sitemap or robots.txt URL, so an attacker controlling that content could push internal-network HTTP URLs into the queue and have them crawled by whichever HTTP client was configured.

2) Nested sitemap fetching bypassed the Request chokepoint entirely

When _XmlSitemapParser encountered <sitemapindex><sitemap><loc>…</loc></sitemap></sitemapindex>, or when RobotsTxtFile.parse_sitemaps forwarded Sitemap: directives into the same pipeline, _fetch_and_process_sitemap dispatched the URL directly to the HTTP client:

async with http_client.stream(
    sitemap_url, 
    method='GET', 
    headers=SITEMAP_HEADERS, 
    proxy_info=proxy_info, 
    timeout=timeout,
) as response:
    ...

No Request was constructed, so the Pydantic validator never ran. Before the fix, the HTTP clients' own send_request() and stream() methods did not call validate_http_url either, so a non-http(s) scheme could pass straight through to the backend client.

The non-HTTP escalation in layer 2 is specific to CurlImpersonateHttpClient, which is backed by curl-cffi / libcurl and speaks gopher, file, dict, ftp, and other non-HTTP protocols. The other clients shipped with Crawlee (HttpxHttpClient, ImpitHttpClient, PlaywrightHttpClient) reject non-http(s) schemes at their own backend layer, regardless of what Crawlee passes in, so they were only affected by layer 1.

Vulnerable paths

Layer 1 — cross-host HTTP (all HTTP clients)

• Source: an attacker-controlled sitemap that lists internal URLs under <urlset><url><loc> or <sitemapindex><sitemap><loc>, or an attacker-controlled robots.txt that lists internal URLs under Sitemap:.
• Sink: the configured HTTP client issues GET requests against those URLs — either via client.request(url=request.url, …) inside crawl() for regular sitemap URLs, or via client.stream(url, …) inside the nested-sitemap fetch.

Layer 2 — non-HTTP schemes (CurlImpersonateHttpClient only)

• Source: a…

Crawlee is a web scraping and browser automation library. From version 1.0.0 to before version 1.7.0, Crawlee is vulnerable to SSRF via sitemap-derived URLs. This issue has been patched in version 1.7.0.

Crawlee is a web scraping and browser automation library. From version 1.0.0 to before version 1.7.0, Crawlee is vulnerable to SSRF via sitemap-derived URLs. This issue has been patched in version 1.7.0.

Overview

• Vulnerability type: Blind SSRF
• Affected components: src/crawlee/_utils/sitemap.py, src/crawlee/_utils/robots.py, src/crawlee/request_loaders/_sitemap_request_loader.py, and all built-in HTTP clients.
• Trigger: an attacker-controlled sitemap or robots.txt containing a URL that points to an internal host (layer 1) or uses a non-http scheme (layer 2).

Two-layer SSRF via sitemap-derived URLs:

1) Cross-host HTTP SSRF

Base case, affects every HTTP client.** Sitemap entries and robots.txt Sitemap: directives were accepted regardless of the host they pointed to. A sitemap on example.com could push http://internal.corp/admin into the crawler's queue, and the configured HTTP client would dispatch the request.

2) Non-HTTP scheme SSRF

Escalation, only CurlImpersonateHttpClient.** Nested-sitemap fetching dispatches the URL straight to the HTTP client, bypassing the Request construction step where Pydantic enforces http(s). Combined with the libcurl-backed CurlImpersonateHttpClient, this lets gopher://, file://, dict://, ftp://, etc., through.

Root cause

Crawlee already validates URL schemes through Pydantic's AnyHttpUrl (via validate_http_url in src/crawlee/_utils/urls.py) wherever a crawl target is materialised as a Request: the Request.url field is declared as Annotated[str, BeforeValidator(validate_http_url), Field(frozen=True)]. Anything that becomes a Request is therefore guaranteed to be http(s).

Two parts of the sitemap pipeline sidestepped this property in different ways:

1) Sitemap-derived URLs were enqueued without any host policy

SitemapRequestLoader took every <urlset><url><loc> entry, wrapped it in Request.from_url (which accepts any valid http(s) URL), and pushed the result into the request queue. RobotsTxtFile.get_sitemaps() returned every Sitemap: directive verbatim. Neither imposed any host check against the parent sitemap or robots.txt URL, so an attacker controlling that content could push internal-network HTTP URLs into the queue and have them crawled by whichever HTTP client was configured.

2) Nested sitemap fetching bypassed the Request chokepoint entirely

When _XmlSitemapParser encountered <sitemapindex><sitemap><loc>…</loc></sitemap></sitemapindex>, or when RobotsTxtFile.parse_sitemaps forwarded Sitemap: directives into the same pipeline, _fetch_and_process_sitemap dispatched the URL directly to the HTTP client:

async with http_client.stream(
    sitemap_url, 
    method='GET', 
    headers=SITEMAP_HEADERS, 
    proxy_info=proxy_info, 
    timeout=timeout,
) as response:
    ...

No Request was constructed, so the Pydantic validator never ran. Before the fix, the HTTP clients' own send_request() and stream() methods did not call validate_http_url either, so a non-http(s) scheme could pass straight through to the backend client.

The non-HTTP escalation in layer 2 is specific to CurlImpersonateHttpClient, which is backed by curl-cffi / libcurl and speaks gopher, file, dict, ftp, and other non-HTTP protocols. The other clients shipped with Crawlee (HttpxHttpClient, ImpitHttpClient, PlaywrightHttpClient) reject non-http(s) schemes at their own backend layer, regardless of what Crawlee passes in, so they were only affected by layer 1.

Vulnerable paths

Layer 1 — cross-host HTTP (all HTTP clients)

• Source: an attacker-controlled sitemap that lists internal URLs under <urlset><url><loc> or <sitemapindex><sitemap><loc>, or an attacker-controlled robots.txt that lists internal URLs under Sitemap:.
• Sink: the configured HTTP client issues GET requests against those URLs — either via client.request(url=request.url, …) inside crawl() for regular sitemap URLs, or via client.stream(url, …) inside the nested-sitemap fetch.

Layer 2 — non-HTTP schemes (CurlImpersonateHttpClient only)

• Source: a…

Overview

• Vulnerability type: Blind SSRF
• Affected components: src/crawlee/_utils/sitemap.py, src/crawlee/_utils/robots.py, src/crawlee/request_loaders/_sitemap_request_loader.py, and all built-in HTTP clients.
• Trigger: an attacker-controlled sitemap or robots.txt containing a URL that points to an internal host (layer 1) or uses a non-http scheme (layer 2).

Two-layer SSRF via sitemap-derived URLs:

1) Cross-host HTTP SSRF

Base case, affects every HTTP client.** Sitemap entries and robots.txt Sitemap: directives were accepted regardless of the host they pointed to. A sitemap on example.com could push http://internal.corp/admin into the crawler's queue, and the configured HTTP client would dispatch the request.

2) Non-HTTP scheme SSRF

Escalation, only CurlImpersonateHttpClient.** Nested-sitemap fetching dispatches the URL straight to the HTTP client, bypassing the Request construction step where Pydantic enforces http(s). Combined with the libcurl-backed CurlImpersonateHttpClient, this lets gopher://, file://, dict://, ftp://, etc., through.

Root cause

Crawlee already validates URL schemes through Pydantic's AnyHttpUrl (via validate_http_url in src/crawlee/_utils/urls.py) wherever a crawl target is materialised as a Request: the Request.url field is declared as Annotated[str, BeforeValidator(validate_http_url), Field(frozen=True)]. Anything that becomes a Request is therefore guaranteed to be http(s).

Two parts of the sitemap pipeline sidestepped this property in different ways:

1) Sitemap-derived URLs were enqueued without any host policy

SitemapRequestLoader took every <urlset><url><loc> entry, wrapped it in Request.from_url (which accepts any valid http(s) URL), and pushed the result into the request queue. RobotsTxtFile.get_sitemaps() returned every Sitemap: directive verbatim. Neither imposed any host check against the parent sitemap or robots.txt URL, so an attacker controlling that content could push internal-network HTTP URLs into the queue and have them crawled by whichever HTTP client was configured.

2) Nested sitemap fetching bypassed the Request chokepoint entirely

When _XmlSitemapParser encountered <sitemapindex><sitemap><loc>…</loc></sitemap></sitemapindex>, or when RobotsTxtFile.parse_sitemaps forwarded Sitemap: directives into the same pipeline, _fetch_and_process_sitemap dispatched the URL directly to the HTTP client:

async with http_client.stream(
    sitemap_url, 
    method='GET', 
    headers=SITEMAP_HEADERS, 
    proxy_info=proxy_info, 
    timeout=timeout,
) as response:
    ...

No Request was constructed, so the Pydantic validator never ran. Before the fix, the HTTP clients' own send_request() and stream() methods did not call validate_http_url either, so a non-http(s) scheme could pass straight through to the backend client.

The non-HTTP escalation in layer 2 is specific to CurlImpersonateHttpClient, which is backed by curl-cffi / libcurl and speaks gopher, file, dict, ftp, and other non-HTTP protocols. The other clients shipped with Crawlee (HttpxHttpClient, ImpitHttpClient, PlaywrightHttpClient) reject non-http(s) schemes at their own backend layer, regardless of what Crawlee passes in, so they were only affected by layer 1.

Vulnerable paths

Layer 1 — cross-host HTTP (all HTTP clients)

• Source: an attacker-controlled sitemap that lists internal URLs under <urlset><url><loc> or <sitemapindex><sitemap><loc>, or an attacker-controlled robots.txt that lists internal URLs under Sitemap:.
• Sink: the configured HTTP client issues GET requests against those URLs — either via client.request(url=request.url, …) inside crawl() for regular sitemap URLs, or via client.stream(url, …) inside the nested-sitemap fetch.

Layer 2 — non-HTTP schemes (CurlImpersonateHttpClient only)

• Source: a…