Vulnerabilities in Heroku’s Build System

Update: Heroku’s official response.

Last week I discovered a major security flaw in the Heroku Cedar stack build system. This vulnerability exposed sensitive information including API keys, private keys and server credentials.

Once I realized the extent of the vulnerability, I immediately informed Heroku. I have been in regular contact with their security team and the problem has since been fixed.

Understanding the issue requires operational knowledge of the Cedar stack build system.

Cedar Build Process

Since Heroku runs on Heroku, after receiving a git push of an application, the build request is dispatched to a regular Heroku app named Codon that handles builds. Codon runs a buildpack which compiles the application so that it can be deployed.

Normally apps running on Heroku are entirely isolated using Linux Containers, but to perform builds Codon runs untrusted code in this container.

Source Code Exposure

I encountered a Ruby exception and backtrace from the Heroku build system while experimenting with custom buildpacks. Ruby backtraces look like this:

app.rb:2:in `foo': undefined method `a' for nil:NilClass (NoMethodError)
        from app.rb:5:in `<main>'

Backtraces include the paths to the source files that encountered the exception. This pointed me to the source files for Codon, which indicated the possibility of gaining read access to the code.

I then ran a custom buildpack that copied the source code into my Heroku app and verified that it was possible to view the source code of Codon.

While examining the source code I discovered that there was another vulnerability that was much more serious than source code exposure.

Sensitive Credential Exposure

Like most Heroku apps, Codon uses environment variables to configure runtime options including sensitive credentials. This ensures that credentials are not checked into version control. However, due to the constraints of Heroku containers, Codon is running as the same user as the buildpack, which is untrusted. This allows the buildpack to dump the environment variables of Codon from the Linux process table:

cat /proc/*/environ

The environment variables exposed included critical credentials such as internal API keys, a SSH private key with access to source code repositories, Redis connection details, and a key with access to their Campfire account.

Disclosure Timeline

Immediately after discovering this vulnerability, I sent an email to Heroku’s security team to start the disclosure process. I requested a PGP key first, as they did not provide one on their website. Here is the discovery and disclosure timeline:

Customer Impact

The build system appears to have been vulnerable since the Cedar stack launched about a year ago. Customer applications and credentials could have been compromised at some point due to the credential exposed by the vulnerability. Anyone who ran applications on Heroku during this period should immediately reset all sensitive credentials, and audit their access logs to determine if any infrastructure or data has been accessed.

I suspect that a variant of this vulnerability may exist in other Platform as a Service build systems. Further research is warranted.

Disclosure: At the time of publishing, I was a Heroku customer. Heroku offered me a paid penetration test contract, but required that I sign a retroactive non-disclosure agreement which would have precluded publishing this article.