CVE-2024-1892

Impact

The following parts of the Scrapy API were found to be vulnerable to a ReDoS attack:

• The XMLFeedSpider class or any subclass that uses the default node iterator: iternodes, as well as direct uses of the scrapy.utils.iterators.xmliter function.

• Scrapy 2.6.0 to 2.11.0: The open_in_browser function for a response without a base tag.

Handling a malicious response could cause extreme CPU and memory usage during the parsing of its content, due to the use of vulnerable regular expressions for that parsing.

Patches

Upgrade to Scrapy 2.11.1.

If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.11.1 is not an option, you may upgrade to Scrapy 1.8.4 instead.

Workarounds

For XMLFeedSpider, switch the node iterator to xml or html.

For open_in_browser, before using the function, either manually review the response content to discard a ReDos attack or manually define the base tag to avoid its automatic definition by open_in_browser later.

Acknowledgements

This security issue was reported by @nicecatch2000 through huntr.com.

A Regular Expression Denial of Service (ReDoS) vulnerability exists in the XMLFeedSpider class of the scrapy/scrapy project, specifically in the parsing of XML content. By crafting malicious XML content that exploits inefficient regular expression complexity used in the parsing process, an attacker can cause a denial-of-service (DoS) condition. This vulnerability allows for the system to hang and consume significant resources, potentially rendering services that utilize Scrapy for XML processing unresponsive.

A Regular Expression Denial of Service (ReDoS) vulnerability exists in the XMLFeedSpider class of the scrapy/scrapy project, specifically in the parsing of XML content. By crafting malicious XML content that exploits inefficient regular expression complexity used in the parsing process, an attacker can cause a denial-of-service (DoS) condition. This vulnerability allows for the system to hang and consume significant resources, potentially rendering services that utilize Scrapy for XML processing unresponsive.

These are all security issues fixed in the python-Scrapy-doc-2.11.1-1.1 package on the GA media of openSUSE Tumbleweed.

It was discovered that Scrapy improperly exposed HTTP authentication credentials to request targets, including during redirects. An attacker could use this issue to gain unauthorized access to user accounts. This issue only affected Ubuntu 18.04 LTS and Ubuntu 20.04 LTS. (CVE-2021-41125)

It was discovered that Scrapy did not remove the cookie header during cross-domain redirects. An attacker could possibly use this issue to gain unauthorized access to user accounts. This issue only affected Ubuntu 18.04 LTS, Ubuntu 20.04 LTS, and Ubuntu 22.04 LTS. (CVE-2022-0577)

It was discovered that Scrapy inefficiently parsed XML content. An attacker could use this issue to cause a denial of service by sending a crafted XML response. This issue only affected Ubuntu 18.04 LTS, Ubuntu 20.04 LTS, and Ubuntu 22.04 LTS. (CVE-2024-1892)

It was discovered that Scrapy did not properly check response size during decompression. An attacker could send a crafted response that would exhaust memory and cause a denial of service. This issue only affected Ubuntu 18.04 LTS, Ubuntu 20.04 LTS, and Ubuntu 22.04 LTS. (CVE-2024-3572)

It was discovered that Scrapy did not remove the authorization header during cross-domain redirects. An attacker could possibly use this issue to gain unauthorized access to user accounts. This issue only affected Ubuntu 18.04 LTS, Ubuntu 20.04 LTS, and Ubuntu 22.04 LTS. (CVE-2024-3574)

It was discovered that Scrapy did not remove the authorization header during redirects that change scheme but remain in the same domain. This issue could possibly be used by an attacker to expose sensitive information or to gain unauthorized access to user accounts. (CVE-2024-1968)