Docker Hub rate limits and Drone Cloud exemption?

Drone Support
#3

I’ve come across the same issue for our on-premise Drone CI and purchased a DockerHub license, using credentials in the drone registry plugin which is running alongside drone-runner-kube. However this doesn’t cover DIND use cases where a user may be using such an image and then performing a docker build, referencing a DockerHub base image. I’ve thought of a couple approaches around this which are not so elegant:

  • Custom private DIND image with read-only credentials so it is auto-authenticated
  • Mount dockerconfig or auth step using a secret in the DIND Drone Service step

Neither of these are ideal and impacting for everyone as they’d need to modify their pipeline steps (several hundred users). I was wondering if you had any suggestions on how this could potentially be handled in the runner to make it non-impacting for users? Possibly via a policy, match on an image (or trusted images) and mounts in readonly credentials?

1 Like
#4

@Kash there are two different scenarios that Drone users need to account for when it comes to the docker plugin and rate limiting.

Scenario 1: Build and Publish to Dockerhub

If you are building and publishing to Dockerhub, you are already providing a username and password to the plugin. The plugins always executes a docker login before it builds and publishes the image. This means that image pulls are going to be authenticated and will receive increased rate limits. If the username and password are associated with a paid dockerhub account, there will be no pull limit.

The recommended solution to rate limiting in this scenario is to ensure the username and password provided to the plugin are associated with a paid account. You could even use the organization secrets feature to provide this at an organization level. However, this may require changing yaml files or changing existing secrets.

Scenario 2: Build and Publish to other Registry, but Dockerfile references Dockerhub Images

If you are supplying a username and password to the dind plugin, but the username and password is for another registry (for example, quay) and if your dockerfile pulls from dockerhub, the pulls would not be authenticated and would be subject to rate limits.

The recommended solution in this scenario is to provide dockerhub credentials using the config parameter. Note that these credentials should be associated with a paid account to avoid rate limiting. The config parameter should contain the contents of a docker/config.json file.

- name: build
  image: plugins/docker
  settings:
    repo: quay.io/foo/bar
    username:
    password:
    config: |
      {
          "auths": {
              "https://index.docker.io/v1/": {
                  "auth": "c3R...zE2"
              }
          }
      }

The config file can also be sourced from a secret:

config:
  from_secret: ...

Global Solution: Using a Global Mirror

Finally, a global solution to this problem is to setup a registry mirror. We have some customers that setup and mirror to proxy and authenticate requests to dockerhub. I am not sure how to configure registry mirrors with kubernetes, but with docker you can configure mirrors by editing the docker daemon.json file.

The docker plugin also provides an option to configure a mirronr in the yaml. You could globally configure this mirror for all pipelines by configuring the following a global environment variable with your runner. This would be the equivalent of setting the mirror in every yaml.

PLUGIN_MIRROR=https://docker.company.com
#5

Thanks Brad for the response.

For a lot of our users they are performing docker image builds detached from the push stage. i.e. build, run and test, and then publishing to a repository is separate.

Based on what you’ve mentioned it looks like we’d have to go with organization secrets, which unfortunately means:

  • Org secret added to each “organization” (we use Gitlab and there are over 100 groups), including any new ones created in the future by users. We could script and automate this.
  • All users update their Drone pipeline dind steps to use credentials from a secret.
#6

I edited my previous comment to discuss registry mirrors as an option. This is the approach most of our customers are using. It requires zero end user changes, but requires setting up a mirror and reverse proxy.

#7

Many thanks Brad, I’ll look at setting up something similar as this seems like the cleanest approach!

#8

@bradrydzewski we solved this in a different way with https://github.com/drone/drone-registry-plugin (which we were already using)

We added an entry for docker.io:

- address: docker.io
  username: dockerhub-user
  password: dockerhub-token

We really needed a solution that doesn’t require all teams to update their .drone.yml files and this seems to be working. Can you see any issues with this fix?

dockerhub-user is a user that we have signed up with a “pro” dockerhub account.

I should mention we are not using Drone cloud, we are running our own Drone instance.

#9

@jimsheldon yes, your solution should work. The only edge case is using our docker-in-docker plugins (plugins/ecr, plugins/gcr) to build and publish to a third party registry (ecr, gcr, etc) but where the Dockerfile pulls images from dockerhub. Does this apply to your setup?

#10

UPDATE, NOV 5, 6:00 PM EST

We have started rolling out changes to our Drone Cloud infrastructure to authenticate image pulls. These changes have been rolled out to our amd64 servers (as of 6pm est). The arm servers and arm64 servers are not updated yet. We will post status updates to this thread when we have more progress to share.

#11

What’s the fix for those of us using the drone autoscaler?

#12

Second question, does DRONE_DOCKER_CONFIG have to be /root/.docker ? We had assumed no.

#13

Thanks for verifying.

We are aware of the issue you’re describing, we are working on a communication for our users around that.

#14

  1. configure the autoscaler to use a custom cloud-init file that writes the config file to the host os

  2. configure the autoscaler to create runners that mount the config file using the DRONE_AGENT_VOLUMES variable

  3. configure the autoscaler to set the DRONE_DOCKER_CONFIG variable using the DRONE_AGENT_ENVIRON variable.

    DRONE_AGENT_ENVIRON=DRONE_DOCKER_CONFIG=/root/.docker/config.json

I will plan a minor release for next week that makes this configuration a little easier. This was previously an uncommon configuration, but now that it is going to be more common, we can optimize and make it easier for everyone. Stay tuned.

correct, it can be any path as long as it points to a file. Note that this file has to be mounted into the runner container in order for the runner to read and parse.

1 Like
#15

We’re seeing an issue with launching new docker runners with the drone-autoscaler. The autoscaler pulls the drone-docker-runner image with an unauthenticated login, and there are no mechanisms to inject credentials to the docker daemon on the host that starts up drone-docker-runner.

I think it would be possible to create an AMI with the credentials pre-baked and signed in, but that seems a very complicated way to solve this.

Would it be possible to add a flag(s) to allow a user to add docker credentials for a docker login?

1 Like
#16

You could use a customized cloud-init file for autoscaler, no need for custom AMI.

#17

the recommended approach would be to customize the cloud-init file to install a ~/.docker/config.json with your credentials. See https://autoscale.drone.io/configure/cloud-init/

EDIT looks like @techknowlogick beat me to this answer :slight_smile:

#18

We have a custom AMI built with /root/.docker/config.json

We have the autoscaler set up to insert this env var (using DRONE_AGENT_ENV_FILE) DRONE_DOCKER_CONFIG=/root/.docker/config.json

The autoscaler has this configuration
DRONE_AGENT_VOLUMES=/root/.docker:/root/.docker

I can check a working agent and see that the runner is working:

56af051cd495 drone/drone-runner-docker:1 "/bin/drone-runner-d…" 32 hours ago Up 32 hours 3000/tcp agent

I can do a docker exec -it /bin/sh into a working drone runner and see DRONE_DOCKER_CONFIG is set correctly and I can read from /root/.docker/config.json

When I use the CLI and type drone server info on a new agent I see:

Error: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

We have validated, today and several times, that the credentials we’re passing to docker works.

Our next step is to set DRONE_RUNNER_VOLUMES to /root/.docker, but we’re running out of options, and the rate limit keeps hitting us. Are we in some edge case with the autoscaler here?

#19

An update on this.

After a lot of frustrated research, wondering why pipelines can pull private repos from docker hub but we’re still hitting a rate limit, we’ve narrowed down why we’re hitting the rate limit to the drone-docker plugin. Because we publish images to another registry (quay) we pull from docker hub and publish to quay - and it looks like drone-docker’s pulls are unauthenticated. We haven’t looked into a fix yet but one might be available. So all our dockerfiles with FROM python/blah for example are being pulled in an unauthenticated way.

We tested this by a multi-stage pipeline where we tried to trick drone into unauthenticated pulls of a private repo, all of which succeeded (meaning the drone step’s docker login worked, which is good) and a drone-docker step that built a container image with a FROM step pointing to a private repo the docker hub credentials we provided should have access to. That failed.

There’s still the lingering worry why we see the autoscaler was making unauthenticated docker hub pulls, even though the credential steps should be satisfied by our build. But maybe we’re doing something wrong there, hard to say.

#20

I discuss this scenario in this comment, as well as possible solutions:

#21

It turns out our fix with https://github.com/drone/drone-registry-plugin didn’t seem to work.

We have since started using /root/.docker/config.json as described in How to prevent DockerHub pull rate limit errors

#22

We have ~55 pipelines with ~200 steps that build using plugin/docker - it’s untenable to simply modify all of those pipelines in a reasonable fashion, and we have no interest or time to build, deploy, and maintain a registry mirror.

Since we can’t modify pipelines or setup a mirror, we are trying to use DRONE_RUNNER_VOLUMES to have drone agents mount the host docker credentials into the pipelines step so that the daemon is already authenticated with dockerhub by the time it starts up. However we are hitting a problem where the variable is somehow defined twice in the agent’s configuration:

We are using a drone autoscaler to launch drone-docker-runner instances in AWS EC2, and providing some additional configuration by defining an agent configuration file using DRONE_AGENT_ENV_FILE.

The env file looks like this:

DRONE_SECRET_SECRET=<secret>
DRONE_SECRET_ENDPOINT=<snip>
DRONE_DOCKER_CONFIG=/root/.docker/config.json
DRONE_RUNNER_VOLUMES=/root/.docker:/root/.docker

All these environment variables are set as expected, but somehow DRONE_RUNNER_VOLUME ends up defined twice in the agent container that starts. The follow is the relevant portion of “docker inspect agent” for one of the auto-scaled agents. Since the empty version of DRONE_RUNNER_VOLUMES is defined second, the value we set is ignored, and the mount does not happen as expected.

            "Env": [
                "DRONE_SECRET_ENDPOINT=<snip>",
                "DRONE_DOCKER_CONFIG=/root/.docker/config.json",
               "DRONE_RUNNER_VOLUMES=/root/.docker:/root/.docker",
                "DRONE_SECRET_SECRET=<snip>",
                "DRONE_RPC_HOST=<snip>",
                "DRONE_RPC_PROTO=https",
                "DRONE_RPC_SERVER=<snip>",
                "DRONE_RPC_SECRET=<snip>",
                "DRONE_RUNNER_CAPACITY=2",
                "DRONE_RUNNER_NAME=agent-unx6C5uD",
                "DRONE_RUNNER_VOLUMES=",
                "DRONE_RUNNER_DEVICES=",
                "DRONE_RUNNER_PRIVILEGED_IMAGES=",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "GODEBUG=netdns=go",
                "DRONE_PLATFORM_OS=linux",
                "DRONE_PLATFORM_ARCH=amd64"
            ],

To be clear, we are a paying customer with a drone enterprise license, and so far the response to this issue has been lackluster. We need a solution, not a work around.