Unauthenticated Remote Code Execution on djangoci.com
Yesterday the Django Security and Operations teams were made aware of a remote code execution vulnerability in the Django Software Foundation's Jenkins infrastructure, used to run tests on the Django code base for GitHub pull requests and release branches. In this blog post, the teams want to outline the course of events.
Impact
The Django Security and Operations teams want to assure that at no point was there any risk about issuing or uploading malicious releases of Django to PyPI or the Django Project website. Official Django releases have always been issued manually by releasers. Neither was there any risk to any user data related to the Django Project website or the Django bug tracker.
Timeline
On May 14th, 2019 at 07:48 UTC the Django Security team was made aware by Ai Ho through its HackerOne project that the Django's Continuous Integration service was susceptible to a remote code execution vulnerability, allowing unauthenticated users to execute arbitrary code.
At 08:01 UTC, the Django Security team acknowledged the report and took immediate steps to mitigate the issue by shutting down the primary Jenkins server. The Jenkins master server was shut down by 08:10 UTC.
At 08:45 UTC, the Operations team started provisioning a new server. In cases of a compromised server, it is almost always impractical to clean it up. Starting with a fresh, clean installation is a considerably better and safer approach.
At 14:59 UTC, the new Jenkins master server was up and running again, with some configuration left to do to get Jenkins jobs working again. About 10 minutes later, at 15:09 UTC, that was the case.
At 15:44 UTC, Jenkins started running tests against GitHub pull requests again.
At 16:00 UTC, the Operations team discussed the necessity of revoking various Let's Encrypt certificates or keys. However, since there was no indication that either the account or the certificate's private key was exposed, it was deemed sufficient to rely on the auto-expiration of the Let's Encrypt certificate. However, a new private key for the djangoci.com certificate was generated during the bootstrapping of the new Jenkins master server.
At 16:50 UTC, the Jenkins Windows nodes were working again and started to process jobs.
General notes regarding security reporting
As always, we ask that potential security issues be reported via private email to security@djangoproject.com or HackerOne, and not via Django's Trac instance or the django-developers list. Please see our security policies for further information.
