I saw your message in Gitter and responded, but wanted to repost in case others come across this thread:
hey all. I am going to create a readonly dockerhub service account and purchase a plan, and then rollout global credentials to our fleet of servers. This will give us unlimited pulls, and seems well worth the $5 per month cost. I expect this to be resolved within the next 24 hours.
I’ve come across the same issue for our on-premise Drone CI and purchased a DockerHub license, using credentials in the drone registry plugin which is running alongside drone-runner-kube. However this doesn’t cover DIND use cases where a user may be using such an image and then performing a docker build, referencing a DockerHub base image. I’ve thought of a couple approaches around this which are not so elegant:
Neither of these are ideal and impacting for everyone as they’d need to modify their pipeline steps (several hundred users). I was wondering if you had any suggestions on how this could potentially be handled in the runner to make it non-impacting for users? Possibly via a policy, match on an image (or trusted images) and mounts in readonly credentials?
@Kash there are two different scenarios that Drone users need to account for when it comes to the docker plugin and rate limiting.
Scenario 1: Build and Publish to Dockerhub
If you are building and publishing to Dockerhub, you are already providing a username and password to the plugin. The plugins always executes a docker login before it builds and publishes the image. This means that image pulls are going to be authenticated and will receive increased rate limits. If the username and password are associated with a paid dockerhub account, there will be no pull limit.
The recommended solution to rate limiting in this scenario is to ensure the username and password provided to the plugin are associated with a paid account. You could even use the organization secrets feature to provide this at an organization level. However, this may require changing yaml files or changing existing secrets.
Scenario 2: Build and Publish to other Registry, but Dockerfile references Dockerhub Images
If you are supplying a username and password to the dind plugin, but the username and password is for another registry (for example, quay) and if your dockerfile pulls from dockerhub, the pulls would not be authenticated and would be subject to rate limits.
The recommended solution in this scenario is to provide dockerhub credentials using the config parameter. Note that these credentials should be associated with a paid account to avoid rate limiting. The config parameter should contain the contents of a docker/config.json file.
- name: build
image: plugins/docker
settings:
repo: quay.io/foo/bar
username:
password:
config: |
{
"auths": {
"https://index.docker.io/v1/": {
"auth": "c3R...zE2"
}
}
}
The config file can also be sourced from a secret:
config:
from_secret: ...
Global Solution: Using a Global Mirror
Finally, a global solution to this problem is to setup a registry mirror. We have some customers that setup and mirror to proxy and authenticate requests to dockerhub. I am not sure how to configure registry mirrors with kubernetes, but with docker you can configure mirrors by editing the docker daemon.json file.
The docker plugin also provides an option to configure a mirronr in the yaml. You could globally configure this mirror for all pipelines by configuring the following a global environment variable with your runner. This would be the equivalent of setting the mirror in every yaml.
PLUGIN_MIRROR=https://docker.company.com
Thanks Brad for the response.
For a lot of our users they are performing docker image builds detached from the push stage. i.e. build, run and test, and then publishing to a repository is separate.
Based on what you’ve mentioned it looks like we’d have to go with organization secrets, which unfortunately means:
I edited my previous comment to discuss registry mirrors as an option. This is the approach most of our customers are using. It requires zero end user changes, but requires setting up a mirror and reverse proxy.
Many thanks Brad, I’ll look at setting up something similar as this seems like the cleanest approach!
@bradrydzewski we solved this in a different way with https://github.com/drone/drone-registry-plugin (which we were already using)
We added an entry for docker.io:
- address: docker.io
username: dockerhub-user
password: dockerhub-token
We really needed a solution that doesn’t require all teams to update their .drone.yml files and this seems to be working. Can you see any issues with this fix?
dockerhub-user is a user that we have signed up with a “pro” dockerhub account.
I should mention we are not using Drone cloud, we are running our own Drone instance.
@jimsheldon yes, your solution should work. The only edge case is using our docker-in-docker plugins (plugins/ecr, plugins/gcr) to build and publish to a third party registry (ecr, gcr, etc) but where the Dockerfile pulls images from dockerhub. Does this apply to your setup?
UPDATE, NOV 5, 6:00 PM EST
We have started rolling out changes to our Drone Cloud infrastructure to authenticate image pulls. These changes have been rolled out to our amd64 servers (as of 6pm est). The arm servers and arm64 servers are not updated yet. We will post status updates to this thread when we have more progress to share.
What’s the fix for those of us using the drone autoscaler?
Second question, does DRONE_DOCKER_CONFIG have to be /root/.docker ? We had assumed no.
Thanks for verifying.
We are aware of the issue you’re describing, we are working on a communication for our users around that.
configure the autoscaler to use a custom cloud-init file that writes the config file to the host os
configure the autoscaler to create runners that mount the config file using the DRONE_AGENT_VOLUMES variable
configure the autoscaler to set the DRONE_DOCKER_CONFIG variable using the DRONE_AGENT_ENVIRON variable.
DRONE_AGENT_ENVIRON=DRONE_DOCKER_CONFIG=/root/.docker/config.json
I will plan a minor release for next week that makes this configuration a little easier. This was previously an uncommon configuration, but now that it is going to be more common, we can optimize and make it easier for everyone. Stay tuned.
correct, it can be any path as long as it points to a file. Note that this file has to be mounted into the runner container in order for the runner to read and parse.
We’re seeing an issue with launching new docker runners with the drone-autoscaler. The autoscaler pulls the drone-docker-runner image with an unauthenticated login, and there are no mechanisms to inject credentials to the docker daemon on the host that starts up drone-docker-runner.
I think it would be possible to create an AMI with the credentials pre-baked and signed in, but that seems a very complicated way to solve this.
Would it be possible to add a flag(s) to allow a user to add docker credentials for a docker login?
You could use a customized cloud-init file for autoscaler, no need for custom AMI.
the recommended approach would be to customize the cloud-init file to install a ~/.docker/config.json with your credentials. See https://autoscale.drone.io/configure/cloud-init/
EDIT looks like @techknowlogick beat me to this answer 
We have a custom AMI built with /root/.docker/config.json
We have the autoscaler set up to insert this env var (using DRONE_AGENT_ENV_FILE) DRONE_DOCKER_CONFIG=/root/.docker/config.json
The autoscaler has this configuration
DRONE_AGENT_VOLUMES=/root/.docker:/root/.docker
I can check a working agent and see that the runner is working:
56af051cd495 drone/drone-runner-docker:1 "/bin/drone-runner-d…" 32 hours ago Up 32 hours 3000/tcp agent
I can do a docker exec -it /bin/sh into a working drone runner and see DRONE_DOCKER_CONFIG is set correctly and I can read from /root/.docker/config.json
When I use the CLI and type drone server info on a new agent I see:
Error: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
We have validated, today and several times, that the credentials we’re passing to docker works.
Our next step is to set DRONE_RUNNER_VOLUMES to /root/.docker, but we’re running out of options, and the rate limit keeps hitting us. Are we in some edge case with the autoscaler here?
An update on this.
After a lot of frustrated research, wondering why pipelines can pull private repos from docker hub but we’re still hitting a rate limit, we’ve narrowed down why we’re hitting the rate limit to the drone-docker plugin. Because we publish images to another registry (quay) we pull from docker hub and publish to quay - and it looks like drone-docker’s pulls are unauthenticated. We haven’t looked into a fix yet but one might be available. So all our dockerfiles with FROM python/blah for example are being pulled in an unauthenticated way.
We tested this by a multi-stage pipeline where we tried to trick drone into unauthenticated pulls of a private repo, all of which succeeded (meaning the drone step’s docker login worked, which is good) and a drone-docker step that built a container image with a FROM step pointing to a private repo the docker hub credentials we provided should have access to. That failed.
There’s still the lingering worry why we see the autoscaler was making unauthenticated docker hub pulls, even though the credential steps should be satisfied by our build. But maybe we’re doing something wrong there, hard to say.
I discuss this scenario in this comment, as well as possible solutions:
It turns out our fix with https://github.com/drone/drone-registry-plugin didn’t seem to work.
We have since started using /root/.docker/config.json as described in How to prevent DockerHub pull rate limit errors