Docker Image Insecurity

Recently while downloading an “official” container image with Docker I saw this line:

ubuntu:14.04: The image you are pulling has been verified

I assumed this referenced Docker’s heavily promoted image signing system and didn’t investigate further at the time. Later, while researching the cryptographic digest system that Docker tries to secure images with, I had the opportunity to explore further. What I found was a total systemic failure of all logic related to image security.

Docker’s report that a downloaded image is “verified” is based solely on the presence of a signed manifest, and Docker never verifies the image checksum from the manifest. An attacker could provide any image alongside a signed manifest. This opens the door to a number of serious vulnerabilities.

Images are downloaded from an HTTPS server and go through an insecure streaming processing pipeline in the Docker daemon:

[decompress] -> [tarsum] -> [unpack]

This pipeline is performant but completely insecure. Untrusted input should not be processed before verifying its signature. Unfortunately Docker processes images three times before checksum verification is supposed to occur.

However, despite Docker’s claims, image checksums are never actually checked. This is the only section⁰ of Docker’s code related to verifying image checksums, and I was unable to trigger the warning even when presenting images with mismatched checksums.

if img.Checksum != "" && img.Checksum != checksum {
  log.Warnf("image layer checksum mismatch: computed %q,
             expected %q", checksum, img.Checksum)
}

Insecure processing pipeline

Decompress

Docker supports three compression algorithms: gzip, bzip2, and xz. The first two use the Go standard library implementations, which are memory-safe, so the exploit types I’d expect to see here are denial of service attacks like crashes and excessive CPU and memory usage.

The third compression algorithm, xz, is more interesting. Since there is no native Go implementation, Docker execs the xz binary to do the decompression.

The xz binary comes from the XZ Utils project, and is built from approximately¹ twenty thousand lines of C code. C is not a memory-safe language. This means malicious input to a C program, in this case the Docker image XZ Utils is unpacking, could potentially execute arbitrary code.

Docker exacerbates this situation by running xz as root. This means that if there is a single vulnerability in xz, a call to docker pull could result in the complete compromise of your entire system.

Tarsum

The use of tarsum is well-meaning but completely flawed. In order to get a deterministic checksum of the contents of an arbitrarily encoded tar file, Docker decodes the tar and then hashes specific portions, while excluding others, in a deterministic order.

Since this processing is done in order to generate the checksum, it is decoding untrusted data which could be designed to exploit the tarsum code². Potential exploits here are denial of service as well as logic flaws that could cause files to be injected, skipped, processed differently, modified, appended to, etc. without the checksum changing.

Unpacking

Unpacking consists of decoding the tar and placing files on the disk. This is extraordinarily dangerous as there have been three other vulnerabilities reported³ in the unpack stage at the time of writing.

There is no situation where data that has not been verified should be unpacked onto disk.

libtrust

libtrust is a Docker package that claims to provide “authorization and access control through a distributed trust graph.” Unfortunately no specification appears to exist, however it looks like it implements some parts of the Javascript Object Signing and Encryption specifications along with other unspecified algorithms.

Downloading an image with a manifest signed and verified using libtrust is what triggers this inaccurate message (only the manifest is checked, not the actual image contents):

ubuntu:14.04: The image you are pulling has been verified

Currently only “official” image manifests published by Docker, Inc are signed using this system, but from discussions I participated in at the last Docker Governance Advisory Board meeting⁴, my understanding is that Docker, Inc is planning on deploying this more widely in the future. The intended goal is centralization with Docker, Inc controlling a Certificate Authority that then signs images and/or client certificates.

I looked for the signing key in Docker’s code but was unable to find it. As it turns out the key is not embedded in the binary as one would expect. Instead the Docker daemon fetches it over HTTPS from a CDN before each image download. This is a terrible approach as a variety of attacks could lead to trusted keys being replaced with malicious ones. These attacks include but are not limited to: compromise of the CDN vendor, compromise of the CDN origin serving the key, and man in the middle attacks on clients downloading the keys.

Remediation

I reported some of the issues I found with the tarsum system before I finished this research, but so far nothing I have reported has been fixed.

Some steps I believe should be taken to improve the security of the Docker image download system:

Drop tarsum and actually verify image digests

Tarsum should not be used for security. Instead, images must be fully downloaded and their cryptographic signatures verified before any processing takes place.

Add privilege isolation

Image processing steps that involve decompression or unpacking should be run in isolated processes (containers?) that have only the bare minimum required privileges to operate. There is no scenario where a decompression tool like xz should be run as root.

Replace libtrust

Libtrust should be replaced with The Update Framework which is explicitly designed to solve the real problems around signing software binaries. The threat model is very comprehensive and addresses many things that have not been considered in libtrust. There is a complete specification as well as a reference implementation written in Python, and I have begun work on a Go implementation and welcome contributions.

As part of adding TUF to Docker, a local keystore should be added that maps root keys to registry URLs so that users can have their own signing keys that are not managed by Docker, Inc.

I would like to note that using non-Docker, Inc hosted registries is a very poor user experience in general. Docker, Inc seems content with relegating third party registries to second class status when there is no technical reason to do so. This is a problem both for the ecosystem in general and the security of end users. A comprehensive, decentralized security model for third party registries is both necessary and desirable. I encourage Docker, Inc to take this into consideration when redesigning their security model and image verification system.

Conclusion

Docker users should be aware that the code responsible for downloading images is shockingly insecure. Users should only download images whose provenance is without question. At present, this does not include “trusted” images hosted by Docker, Inc including the official Ubuntu and other base images.

The best option is to block index.docker.io locally, and download and verify images manually before importing them into Docker using docker load. Red Hat’s security blog has a good post about this.

Thanks to Lewis Marshall for pointing out the tarsums are never verified.

SMS Vulnerability in Twitter, Facebook, and Venmo

Update: Twitter has fixed the issue for users of short codes. Users that use a “long code” should enable the PIN code in their account.

Twitter users with SMS enabled are vulnerable to an attack that allows anyone to post to their account. The attacker only needs knowledge of the mobile number associated with a target’s Twitter account. Messages can then be sent to Twitter with the source number spoofed.

Like email, the originating address of a SMS cannot be trusted. Many SMS gateways allow the originating address of a message to be set to an arbitrary identifier, including someone else’s number.

Facebook and Venmo were also vulnerable to the same spoofing attack, but the issues were resolved after disclosing to their respective security teams.

Scope

Users

Users of Twitter that have a mobile number associated with their account and have not set a PIN code are vulnerable. All of the Twitter SMS commands can be used by an attacker, including the ability to post tweets and modify profile info.

Service Providers

All services that trust the originating address of SMS messages implicitly and are not using a short code are vulnerable.

Mitigation

Users

Until Twitter removes the ability to post via non-short code numbers, users should enable PIN codes (if available in their region) or disable the mobile text messaging feature.

Twitter has a PIN code feature that requires every message to be prepended with a four-digit alphanumeric code. This feature mitigates the issue, but is not available to users inside the United States.

Service Providers

The cleanest solution for providers is to use only an SMS short code to receive incoming messages. In most cases, messages to short codes do not leave the carrier network and can only be sent by subscribers. This removes the ease of spoofing via SMS gateways.

An alternative, less user-friendly but more secure solution is to require a challenge-response for every message. After receiving an SMS, the service would reply with a short alphanumeric string that needs to be repeated back before the message is processed.

Disclosure Timelines

Twitter

The issue I filed was initially inspected by a member of their security team, but was then routed to the normal support team who did not believe that SMS spoofing was possible. I then reached out directly to someone on the security team who said that it was an “old issue” but that they did not want me to publish until they got “a fix in place”. I received no further communication from Twitter.

Facebook

Initially Facebook did not respond to my report on their security vulnerability page. I then emailed a friend who works at Facebook, who facilitated my contact with their security team.

Disclosure: I will receive a bounty from Facebook for finding and reporting this issue to them. The Facebook bounty program requires responsible disclosure and time to resolve internally in “good faith” before publishing.

Venmo

I initially disclosed this issue to Venmo support, as they do not have a security contact published. When I did not receive a response, I notified the Braintree security team (Braintree recently acquired Venmo), who responded very promptly.

Vulnerabilities in Heroku’s Build System

Update: Heroku’s official response.

Last week I discovered a major security flaw in the Heroku Cedar stack build system. This vulnerability exposed sensitive information including API keys, private keys and server credentials.

Once I realized the extent of the vulnerability, I immediately informed Heroku. I have been in regular contact with their security team and the problem has since been fixed.

Understanding the issue requires operational knowledge of the Cedar stack build system.

Cedar Build Process

Since Heroku runs on Heroku, after receiving a git push of an application, the build request is dispatched to a regular Heroku app named Codon that handles builds. Codon runs a buildpack which compiles the application so that it can be deployed.

Normally apps running on Heroku are entirely isolated using Linux Containers, but to perform builds Codon runs untrusted code in this container.

Source Code Exposure

I encountered a Ruby exception and backtrace from the Heroku build system while experimenting with custom buildpacks. Ruby backtraces look like this:

app.rb:2:in `foo': undefined method `a' for nil:NilClass (NoMethodError)
        from app.rb:5:in `<main>'

Backtraces include the paths to the source files that encountered the exception. This pointed me to the source files for Codon, which indicated the possibility of gaining read access to the code.

I then ran a custom buildpack that copied the source code into my Heroku app and verified that it was possible to view the source code of Codon.

While examining the source code I discovered that there was another vulnerability that was much more serious than source code exposure.

Sensitive Credential Exposure

Like most Heroku apps, Codon uses environment variables to configure runtime options including sensitive credentials. This ensures that credentials are not checked into version control. However, due to the constraints of Heroku containers, Codon is running as the same user as the buildpack, which is untrusted. This allows the buildpack to dump the environment variables of Codon from the Linux process table:

cat /proc/*/environ

The environment variables exposed included critical credentials such as internal API keys, a SSH private key with access to source code repositories, Redis connection details, and a key with access to their Campfire account.

Disclosure Timeline

Immediately after discovering this vulnerability, I sent an email to Heroku’s security team to start the disclosure process. I requested a PGP key first, as they did not provide one on their website. Here is the discovery and disclosure timeline:

Customer Impact

The build system appears to have been vulnerable since the Cedar stack launched about a year ago. Customer applications and credentials could have been compromised at some point due to the credential exposed by the vulnerability. Anyone who ran applications on Heroku during this period should immediately reset all sensitive credentials, and audit their access logs to determine if any infrastructure or data has been accessed.

I suspect that a variant of this vulnerability may exist in other Platform as a Service build systems. Further research is warranted.

Disclosure: At the time of publishing, I was a Heroku customer. Heroku offered me a paid penetration test contract, but required that I sign a retroactive non-disclosure agreement which would have precluded publishing this article.