CVE-2024-3572

Impact

Scrapy limits allowed response sizes by default through the DOWNLOAD_MAXSIZE and DOWNLOAD_WARNSIZE settings.

However, those limits were only being enforced during the download of the raw, usually-compressed response bodies, and not during decompression, making Scrapy vulnerable to decompression bombs.

A malicious website being scraped could send a small response that, on decompression, could exhaust the memory available to the Scrapy process, potentially affecting any other process sharing that memory, and affecting disk usage in case of uncompressed response caching.

Patches

Upgrade to Scrapy 2.11.1.

If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.11.1 is not an option, you may upgrade to Scrapy 1.8.4 instead.

Workarounds

There is no easy workaround.

Disabling HTTP decompression altogether is impractical, as HTTP compression is a rather common practice.

However, it is technically possible to manually backport the 2.11.1 or 1.8.4 fix, replacing the corresponding components of an unpatched version of Scrapy with patched versions copied into your own code.

Acknowledgements

This security issue was reported by @dmandefy through huntr.com.

Impact

Scrapy limits allowed response sizes by default through the DOWNLOAD_MAXSIZE and DOWNLOAD_WARNSIZE settings.

However, those limits were only being enforced during the download of the raw, usually-compressed response bodies, and not during decompression, making Scrapy vulnerable to decompression bombs.

A malicious website being scraped could send a small response that, on decompression, could exhaust the memory available to the Scrapy process, potentially affecting any other process sharing that memory, and affecting disk usage in case of uncompressed response caching.

Patches

Upgrade to Scrapy 2.11.1.

If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.11.1 is not an option, you may upgrade to Scrapy 1.8.4 instead.

Workarounds

There is no easy workaround.

Disabling HTTP decompression altogether is impractical, as HTTP compression is a rather common practice.

However, it is technically possible to manually backport the 2.11.1 or 1.8.4 fix, replacing the corresponding components of an unpatched version of Scrapy with patched versions copied into your own code.

Acknowledgements

This security issue was reported by @dmandefy through huntr.com.

The scrapy/scrapy project is vulnerable to XML External Entity (XXE) attacks due to the use of lxml.etree.fromstring for parsing untrusted XML data without proper validation. This vulnerability allows attackers to perform denial of service attacks, access local files, generate network connections, or circumvent firewalls by submitting specially crafted XML data.

It was discovered that Scrapy improperly exposed HTTP authentication credentials to request targets, including during redirects. An attacker could use this issue to gain unauthorized access to user accounts. This issue only affected Ubuntu 18.04 LTS and Ubuntu 20.04 LTS. (CVE-2021-41125)

It was discovered that Scrapy did not remove the cookie header during cross-domain redirects. An attacker could possibly use this issue to gain unauthorized access to user accounts. This issue only affected Ubuntu 18.04 LTS, Ubuntu 20.04 LTS, and Ubuntu 22.04 LTS. (CVE-2022-0577)

It was discovered that Scrapy inefficiently parsed XML content. An attacker could use this issue to cause a denial of service by sending a crafted XML response. This issue only affected Ubuntu 18.04 LTS, Ubuntu 20.04 LTS, and Ubuntu 22.04 LTS. (CVE-2024-1892)

It was discovered that Scrapy did not properly check response size during decompression. An attacker could send a crafted response that would exhaust memory and cause a denial of service. This issue only affected Ubuntu 18.04 LTS, Ubuntu 20.04 LTS, and Ubuntu 22.04 LTS. (CVE-2024-3572)

It was discovered that Scrapy did not remove the authorization header during cross-domain redirects. An attacker could possibly use this issue to gain unauthorized access to user accounts. This issue only affected Ubuntu 18.04 LTS, Ubuntu 20.04 LTS, and Ubuntu 22.04 LTS. (CVE-2024-3574)

It was discovered that Scrapy did not remove the authorization header during redirects that change scheme but remain in the same domain. This issue could possibly be used by an attacker to expose sensitive information or to gain unauthorized access to user accounts. (CVE-2024-1968)