CVE-2024-3574

Impact

When you send a request with the Authorization header to one domain, and the response asks to redirect to a different domain, Scrapy’s built-in redirect middleware creates a follow-up redirect request that keeps the original Authorization header, leaking its content to that second domain.

The right behavior would be to drop the Authorization header instead, in this scenario.

Patches

Upgrade to Scrapy 2.11.1.

If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.11.1 is not an option, you may upgrade to Scrapy 1.8.4 instead.

Workarounds

If you cannot upgrade, make sure that you are not using the Authentication header, either directly or through some third-party plugin.

If you need to use that header in some requests, add "dont_redirect": True to the request.meta dictionary of those requests to disable following redirects for them.

If you need to keep (same domain) redirect support on those requests, make sure you trust the target website not to redirect your requests to a different domain.

Acknowledgements

This security issue was reported by @ranjit-git through huntr.com.

Impact

When you send a request with the Authorization header to one domain, and the response asks to redirect to a different domain, Scrapy’s built-in redirect middleware creates a follow-up redirect request that keeps the original Authorization header, leaking its content to that second domain.

The right behavior would be to drop the Authorization header instead, in this scenario.

Patches

Upgrade to Scrapy 2.11.1.

If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.11.1 is not an option, you may upgrade to Scrapy 1.8.4 instead.

Workarounds

If you cannot upgrade, make sure that you are not using the Authentication header, either directly or through some third-party plugin.

If you need to use that header in some requests, add "dont_redirect": True to the request.meta dictionary of those requests to disable following redirects for them.

If you need to keep (same domain) redirect support on those requests, make sure you trust the target website not to redirect your requests to a different domain.

Acknowledgements

This security issue was reported by @ranjit-git through huntr.com.

In scrapy version 2.10.1, an issue was identified where the Authorization header, containing credentials for server authentication, is leaked to a third-party site during a cross-domain redirect. This vulnerability arises from the failure to remove the Authorization header when redirecting across domains. The exposure of the Authorization header to unauthorized actors could potentially allow for account hijacking.

It was discovered that Scrapy improperly exposed HTTP authentication credentials to request targets, including during redirects. An attacker could use this issue to gain unauthorized access to user accounts. This issue only affected Ubuntu 18.04 LTS and Ubuntu 20.04 LTS. (CVE-2021-41125)

It was discovered that Scrapy did not remove the cookie header during cross-domain redirects. An attacker could possibly use this issue to gain unauthorized access to user accounts. This issue only affected Ubuntu 18.04 LTS, Ubuntu 20.04 LTS, and Ubuntu 22.04 LTS. (CVE-2022-0577)

It was discovered that Scrapy inefficiently parsed XML content. An attacker could use this issue to cause a denial of service by sending a crafted XML response. This issue only affected Ubuntu 18.04 LTS, Ubuntu 20.04 LTS, and Ubuntu 22.04 LTS. (CVE-2024-1892)

It was discovered that Scrapy did not properly check response size during decompression. An attacker could send a crafted response that would exhaust memory and cause a denial of service. This issue only affected Ubuntu 18.04 LTS, Ubuntu 20.04 LTS, and Ubuntu 22.04 LTS. (CVE-2024-3572)

It was discovered that Scrapy did not remove the authorization header during cross-domain redirects. An attacker could possibly use this issue to gain unauthorized access to user accounts. This issue only affected Ubuntu 18.04 LTS, Ubuntu 20.04 LTS, and Ubuntu 22.04 LTS. (CVE-2024-3574)

It was discovered that Scrapy did not remove the authorization header during redirects that change scheme but remain in the same domain. This issue could possibly be used by an attacker to expose sensitive information or to gain unauthorized access to user accounts. (CVE-2024-1968)