CVE-2021-41125
HTTP authentication credential leak to target websites in scrapy
Scrapy is a high-level web crawling and scraping framework for Python. If you use HttpAuthMiddleware (i.e. the http_user and http_pass spider attributes) for HTTP authentication, all requests will expose your credentials to the request target. This includes requests generated by Scrapy components, such as robots.txt requests sent by Scrapy when the ROBOTSTXT_OBEY setting is set to True, or as requests reached through redirects. Upgrade to Scrapy 2.5.1 and use the new http_auth_domain spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead. If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basic_auth_header function to convert your credentials into a value that you can assign to the Authorization header of your request, instead of defining your credentials globally using HttpAuthMiddleware.
Attack profile
How easy it is to exploit
- Attack vector
- Network
- Complexity
- Low
- Privileges needed
- Low
- User interaction
- Required
What an attacker can affect
- Confidentiality
- High
- Integrity
- None
- Availability
- None
Exploit likelihood
What each source reports
CVE List Score used
HTTP authentication credential leak to target websites in scrapy
Scrapy is a high-level web crawling and scraping framework for Python. If you use HttpAuthMiddleware (i.e. the http_user and http_pass spider attributes) for HTTP authentication, all requests will expose your credentials to the request target. This includes requests generated by Scrapy components, such as robots.txt requests sent by Scrapy when the ROBOTSTXT_OBEY setting is set to True, or as requests reached through redirects. Upgrade to Scrapy 2.5.1 and use the new http_auth_domain spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead. If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basic_auth_header function to convert your credentials into a value that you can assign to the Authorization header of your request, instead of defining your credentials globally using HttpAuthMiddleware.
- Record
CVE-2021-41125- CVSS 3.1
- 5.7 Medium
CVSS:3.1/AV:N/AC:L/PR:L/UI:R/S:U/C:H/I:N/A:N - Weakness
- CWE-200
- Updated
- 2026-11-02
EUVDEUVD-2021-0239
Scrapy is a high-level web crawling and scraping framework for Python. If you use HttpAuthMiddleware (i.e. the http_user and http_pass spider attributes) for HTTP authentication, all requests will expose your credentials to the request target. This includes requests generated by Scrapy components, such as robots.txt requests sent by Scrapy when the ROBOTSTXT_OBEY setting is set to True, or as requests reached through redirects. Upgrade to Scrapy 2.5.1 and use the new http_auth_domain spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead. If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basic_auth_header function to convert your credentials into a value that you can assign to the Authorization header of your request, instead of defining your credentials globally using HttpAuthMiddleware.
- Record
EUVD-2021-0239- CVSS 3.1
- 5.7 Medium
CVSS:3.1/AV:N/AC:L/PR:L/UI:R/S:U/C:H/I:N/A:N - Updated
- 2026-11-03
GitHub AdvisoriesGHSA-jwqp-28gf-p498
Scrapy HTTP authentication credentials potentially leaked to target websites
Impact
If you use HttpAuthMiddleware (i.e. the http_user and http_pass spider attributes) for HTTP authentication, all requests will expose your credentials to the request target.
This includes requests generated by Scrapy components, such as robots.txt requests sent by Scrapy when the ROBOTSTXT_OBEY setting is set to True, or as requests reached through redirects.
Patches
Upgrade to Scrapy 2.5.1 and use the new http_auth_domain spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials.
If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead.
Workarounds
If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basic_auth_header function to convert your credentials into a value that you can assign to the Authorization header of your request, instead of defining your credentials globally using HttpAuthMiddleware.
For more information
If you have any questions or comments about this advisory:
- Record
GHSA-jwqp-28gf-p498- CVSS 4.0
- 6.9 Medium
CVSS:4.0/AV:N/AC:L/AT:N/PR:L/UI:P/VC:H/VI:N/VA:N/SC:N/SI:N/SA:N - CVSS 3.1
- 5.7 Medium
CVSS:3.1/AV:N/AC:L/PR:L/UI:R/S:U/C:H/I:N/A:N - Weakness
- CWE-200, CWE-522
- Updated
- 2026-11-02
NVD
Scrapy is a high-level web crawling and scraping framework for Python. If you use HttpAuthMiddleware (i.e. the http_user and http_pass spider attributes) for HTTP authentication, all requests will expose your credentials to the request target. This includes requests generated by Scrapy components, such as robots.txt requests sent by Scrapy when the ROBOTSTXT_OBEY setting is set to True, or as requests reached through redirects. Upgrade to Scrapy 2.5.1 and use the new http_auth_domain spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead. If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basic_auth_header function to convert your credentials into a value that you can assign to the Authorization header of your request, instead of defining your credentials globally using HttpAuthMiddleware.
- Record
CVE-2021-41125- CVSS 3.1
- 5.7 Medium
CVSS:3.1/AV:N/AC:L/PR:L/UI:R/S:U/C:H/I:N/A:N - CVSS 3.1
- 6.5 Medium
CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:N/A:N - CVSS 2.0
- 4.0 Medium
AV:N/AC:L/Au:S/C:P/I:N/A:N - Weakness
- CWE-200, CWE-522
- Updated
- 2026-11-02
OSV
Scrapy is a high-level web crawling and scraping framework for Python. If you use HttpAuthMiddleware (i.e. the http_user and http_pass spider attributes) for HTTP authentication, all requests will expose your credentials to the request target. This includes requests generated by Scrapy components, such as robots.txt requests sent by Scrapy when the ROBOTSTXT_OBEY setting is set to True, or as requests reached through redirects. Upgrade to Scrapy 2.5.1 and use the new http_auth_domain spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead. If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basic_auth_header function to convert your credentials into a value that you can assign to the Authorization header of your request, instead of defining your credentials globally using HttpAuthMiddleware.
- Record
CVE-2021-41125- CVSS 3.1
- 6.5 Medium
CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:N/A:N - Updated
- 2026-11-02
OSVDEBIAN-CVE-2021-41125
Scrapy is a high-level web crawling and scraping framework for Python. If you use HttpAuthMiddleware (i.e. the http_user and http_pass spider attributes) for HTTP authentication, all requests will expose your credentials to the request target. This includes requests generated by Scrapy components, such as robots.txt requests sent by Scrapy when the ROBOTSTXT_OBEY setting is set to True, or as requests reached through redirects. Upgrade to Scrapy 2.5.1 and use the new http_auth_domain spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead. If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basic_auth_header function to convert your credentials into a value that you can assign to the Authorization header of your request, instead of defining your credentials globally using HttpAuthMiddleware.
- Record
DEBIAN-CVE-2021-41125- CVSS 3.1
- 6.5 Medium
CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:N/A:N - Updated
- 2026-11-02
Show 5 moreShow less
OSVGHSA-jwqp-28gf-p498
Scrapy HTTP authentication credentials potentially leaked to target websites
Impact
If you use HttpAuthMiddleware (i.e. the http_user and http_pass spider attributes) for HTTP authentication, all requests will expose your credentials to the request target.
This includes requests generated by Scrapy components, such as robots.txt requests sent by Scrapy when the ROBOTSTXT_OBEY setting is set to True, or as requests reached through redirects.
Patches
Upgrade to Scrapy 2.5.1 and use the new http_auth_domain spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials.
If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead.
Workarounds
If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basic_auth_header function to convert your credentials into a value that you can assign to the Authorization header of your request, instead of defining your credentials globally using HttpAuthMiddleware.
For more information
If you have any questions or comments about this advisory:
- Record
GHSA-jwqp-28gf-p498- CVSS 4.0
- 6.9 Medium
CVSS:4.0/AV:N/AC:L/AT:N/PR:L/UI:P/VC:H/VI:N/VA:N/SC:N/SI:N/SA:N - CVSS 3.1
- 5.7 Medium
CVSS:3.1/AV:N/AC:L/PR:L/UI:R/S:U/C:H/I:N/A:N - Weakness
- CWE-200, CWE-522
- Updated
- 2026-11-02
OSVUBUNTU-CVE-2021-41125
Scrapy is a high-level web crawling and scraping framework for Python. If you use HttpAuthMiddleware (i.e. the http_user and http_pass spider attributes) for HTTP authentication, all requests will expose your credentials to the request target. This includes requests generated by Scrapy components, such as robots.txt requests sent by Scrapy when the ROBOTSTXT_OBEY setting is set to True, or as requests reached through redirects. Upgrade to Scrapy 2.5.1 and use the new http_auth_domain spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead. If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basic_auth_header function to convert your credentials into a value that you can assign to the Authorization header of your request, instead of defining your credentials globally using HttpAuthMiddleware.
- Record
UBUNTU-CVE-2021-41125- CVSS 3.1
- 5.7 Medium
CVSS:3.1/AV:N/AC:L/PR:L/UI:R/S:U/C:H/I:N/A:N - CVSS 3.1
- 6.5 Medium
CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:N/A:N - Updated
- 2026-11-02
OSVopenSUSE-SU-2024:11558-1 No score
python-Scrapy-doc-2.5.1-1.1 on GA media
These are all security issues fixed in the python-Scrapy-doc-2.5.1-1.1 package on the GA media of openSUSE Tumbleweed.
- Record
openSUSE-SU-2024:11558-1- Updated
- 2026-11-02
OSVPYSEC-2021-363 No score
Scrapy is a high-level web crawling and scraping framework for Python. If you use HttpAuthMiddleware (i.e. the http_user and http_pass spider attributes) for HTTP authentication, all requests will expose your credentials to the request target. This includes requests generated by Scrapy components, such as robots.txt requests sent by Scrapy when the ROBOTSTXT_OBEY setting is set to True, or as requests reached through redirects. Upgrade to Scrapy 2.5.1 and use the new http_auth_domain spider attribute to control which domains are allowed to receive the configured HTTP authentication credentials. If you are using Scrapy 1.8 or a lower version, and upgrading to Scrapy 2.5.1 is not an option, you may upgrade to Scrapy 1.8.1 instead. If you cannot upgrade, set your HTTP authentication credentials on a per-request basis, using for example the w3lib.http.basic_auth_header function to convert your credentials into a value that you can assign to the Authorization header of your request, instead of defining your credentials globally using HttpAuthMiddleware.
- Record
PYSEC-2021-363- Updated
- 2026-11-02
OSVUSN-7476-1 No score
python-scrapy vulnerabilities
It was discovered that Scrapy improperly exposed HTTP authentication credentials to request targets, including during redirects. An attacker could use this issue to gain unauthorized access to user accounts. This issue only affected Ubuntu 18.04 LTS and Ubuntu 20.04 LTS. (CVE-2021-41125)
It was discovered that Scrapy did not remove the cookie header during cross-domain redirects. An attacker could possibly use this issue to gain unauthorized access to user accounts. This issue only affected Ubuntu 18.04 LTS, Ubuntu 20.04 LTS, and Ubuntu 22.04 LTS. (CVE-2022-0577)
It was discovered that Scrapy inefficiently parsed XML content. An attacker could use this issue to cause a denial of service by sending a crafted XML response. This issue only affected Ubuntu 18.04 LTS, Ubuntu 20.04 LTS, and Ubuntu 22.04 LTS. (CVE-2024-1892)
It was discovered that Scrapy did not properly check response size during decompression. An attacker could send a crafted response that would exhaust memory and cause a denial of service. This issue only affected Ubuntu 18.04 LTS, Ubuntu 20.04 LTS, and Ubuntu 22.04 LTS. (CVE-2024-3572)
It was discovered that Scrapy did not remove the authorization header during cross-domain redirects. An attacker could possibly use this issue to gain unauthorized access to user accounts. This issue only affected Ubuntu 18.04 LTS, Ubuntu 20.04 LTS, and Ubuntu 22.04 LTS. (CVE-2024-3574)
It was discovered that Scrapy did not remove the authorization header during redirects that change scheme but remain in the same domain. This issue could possibly be used by an attacker to expose sensitive information or to gain unauthorized access to user accounts. (CVE-2024-1968)
- Record
USN-7476-1- Updated
- 2026-11-02
Affected products
Affected packages
| Package | Affected | Fixed |
|---|---|---|
scrapyPyPI |
before 1.8.1 | 1.8.1 |
scrapyPyPI |
2.0.0 up to 2.5.1 | 2.5.1 |
python-scrapyDebian:12 |
before 2.5.1-1 | 2.5.1-1 |
python-scrapyDebian:13 |
before 2.5.1-1 | 2.5.1-1 |
python-scrapyDebian:14 |
before 2.5.1-1 | 2.5.1-1 |
python-scrapyDebian:9 From an advisory covering 2 CVEs |
before 1.0.3-2+deb9u1, plus listed versions 1.0.3-2 |
1.0.3-2+deb9u1 |
scrapyPyPI |
before 1.8.1, plus listed versions 0.10.4.23640.12.0.25500.14.10.14.20.14.3 Show 45 moreShow less0.14.40.16.00.16.10.16.20.16.30.16.40.16.50.18.00.18.10.18.20.18.30.18.40.20.00.20.10.20.20.22.00.22.10.22.20.24.00.24.10.24.20.24.30.24.40.24.50.24.60.70.80.91.0.01.0.0rc11.0.0rc21.0.0rc31.0.11.0.21.0.31.0.41.0.51.0.61.0.71.1.01.1.0rc11.1.0rc21.1.0rc31.1.0rc41.1.1 |
1.8.1 |
scrapyPyPI |
2.0.0 up to 2.5.1, plus listed versions 2.0.12.1.02.2.02.2.12.3.02.4.02.4.12.5.0 |
2.5.1 |
scrapyPyPI |
before 1.8.1, 2.0.0 up to 2.5.1, plus listed versions 0.10.4.23640.12.0.25500.14.10.14.20.14.3 Show 45 moreShow less0.14.40.16.00.16.10.16.20.16.30.16.40.16.50.18.00.18.10.18.20.18.30.18.40.20.00.20.10.20.20.22.00.22.10.22.20.24.00.24.10.24.20.24.30.24.40.24.50.24.60.70.80.91.0.01.0.0rc11.0.0rc21.0.0rc31.0.11.0.21.0.31.0.41.0.51.0.61.0.71.1.01.1.0rc11.1.0rc21.1.0rc31.1.0rc41.1.1 |
1.8.12.5.1 |
python-scrapyUbuntu:16.04:LTS |
All versions, plus listed versions 1.0.0-11.0.3-1 |
None known |
Show 7 moreShow less
| Package | Affected | Fixed |
|---|---|---|
python-scrapyUbuntu:Pro:18.04:LTS |
before 1.5.0-1ubuntu0.1~esm1, plus listed versions 1.3.0-1~exp21.4.0-11.5.0-1 |
1.5.0-1ubuntu0.1~esm1 |
python-scrapyUbuntu:Pro:20.04:LTS |
before 1.7.3-1ubuntu0.1~esm1, plus listed versions 1.7.3-1 |
1.7.3-1ubuntu0.1~esm1 |
python-scrapyUbuntu:Pro:18.04:LTS From an advisory covering 6 CVEs |
before 1.5.0-1ubuntu0.1~esm1, plus listed versions 1.3.0-1~exp21.4.0-11.5.0-1 |
1.5.0-1ubuntu0.1~esm1 |
python-scrapyUbuntu:Pro:20.04:LTS From an advisory covering 6 CVEs |
before 1.7.3-1ubuntu0.1~esm1, plus listed versions 1.7.3-1 |
1.7.3-1ubuntu0.1~esm1 |
python-scrapyUbuntu:Pro:22.04:LTS From an advisory covering 6 CVEs |
before 2.5.1-2ubuntu0.1~esm1, plus listed versions 2.4.1-22.5.1-2 |
2.5.1-2ubuntu0.1~esm1 |
python-scrapyUbuntu:Pro:24.04:LTS From an advisory covering 6 CVEs |
before 2.11.1-1ubuntu0.1~esm2, plus listed versions 2.10.0-12.11.0-12.11.0-22.11.1-1 |
2.11.1-1ubuntu0.1~esm2 |
python-ScrapyopenSUSE:Tumbleweed |
before 2.5.1-1.1 | 2.5.1-1.1 |
History
| Recorded | Changed | Sources |
|---|---|---|
| affected, facts | ||
| aliases | EUVD | |
| first indexed | CVE List, GitHub Advisories, NVD, OSV |
Dates are when Every CVE recorded each change. Upstream dates may differ.
In short
CVE-2021-41125 is a medium vulnerability (CVSS 5.7) in Scrapy, published October 6, 2021. No source lists public exploit code. EPSS puts the chance of exploitation in the next 30 days at 1.26%, the 69th percentile. Fixed in scrapy 1.8.1.
References
github.com/scrapy/scrapy/security/advisories/GHSA-jwqp-28gf-p498
w3lib.readthedocs.io/en/latest/w3lib.html
doc.scrapy.org/en/latest/topics/downloader-middleware.html
lists.debian.org/debian-lts-announce/2022/03/msg00021.html
nvd.nist.gov/vuln/detail/CVE-2021-41125
Show 13 moreShow less
security-tracker.debian.org/tracker/CVE-2021-41125
ubuntu.com/security/CVE-2021-41125
cve.org/CVERecord?id=CVE-2021-41125
suse.com/security/cve/CVE-2021-41125
Left out here: 52 versions. The JSON record has them.