I edited my previous comment to discuss registry mirrors as an option. This is the approach most of our customers are using. It requires zero end user changes, but requires setting up a mirror and reverse proxy.
Many thanks Brad, I’ll look at setting up something similar as this seems like the cleanest approach!
@bradrydzewski we solved this in a different way with https://github.com/drone/drone-registry-plugin (which we were already using)
We added an entry for docker.io:
- address: docker.io
username: dockerhub-user
password: dockerhub-token
We really needed a solution that doesn’t require all teams to update their .drone.yml files and this seems to be working. Can you see any issues with this fix?
dockerhub-user is a user that we have signed up with a “pro” dockerhub account.
I should mention we are not using Drone cloud, we are running our own Drone instance.
@jimsheldon yes, your solution should work. The only edge case is using our docker-in-docker plugins (plugins/ecr, plugins/gcr) to build and publish to a third party registry (ecr, gcr, etc) but where the Dockerfile pulls images from dockerhub. Does this apply to your setup?
UPDATE, NOV 5, 6:00 PM EST
We have started rolling out changes to our Drone Cloud infrastructure to authenticate image pulls. These changes have been rolled out to our amd64 servers (as of 6pm est). The arm servers and arm64 servers are not updated yet. We will post status updates to this thread when we have more progress to share.
What’s the fix for those of us using the drone autoscaler?
Second question, does DRONE_DOCKER_CONFIG have to be /root/.docker ? We had assumed no.
Thanks for verifying.
We are aware of the issue you’re describing, we are working on a communication for our users around that.
-
configure the autoscaler to use a custom cloud-init file that writes the config file to the host os
-
configure the autoscaler to create runners that mount the config file using the DRONE_AGENT_VOLUMES variable
-
configure the autoscaler to set the
DRONE_DOCKER_CONFIGvariable using the DRONE_AGENT_ENVIRON variable.DRONE_AGENT_ENVIRON=DRONE_DOCKER_CONFIG=/root/.docker/config.json
I will plan a minor release for next week that makes this configuration a little easier. This was previously an uncommon configuration, but now that it is going to be more common, we can optimize and make it easier for everyone. Stay tuned.
correct, it can be any path as long as it points to a file. Note that this file has to be mounted into the runner container in order for the runner to read and parse.
1 Like
We’re seeing an issue with launching new docker runners with the drone-autoscaler. The autoscaler pulls the drone-docker-runner image with an unauthenticated login, and there are no mechanisms to inject credentials to the docker daemon on the host that starts up drone-docker-runner.
I think it would be possible to create an AMI with the credentials pre-baked and signed in, but that seems a very complicated way to solve this.
Would it be possible to add a flag(s) to allow a user to add docker credentials for a docker login?
1 Like
You could use a customized cloud-init file for autoscaler, no need for custom AMI.
the recommended approach would be to customize the cloud-init file to install a ~/.docker/config.json with your credentials. See https://autoscale.drone.io/configure/cloud-init/
EDIT looks like @techknowlogick beat me to this answer 
We have a custom AMI built with /root/.docker/config.json
We have the autoscaler set up to insert this env var (using DRONE_AGENT_ENV_FILE) DRONE_DOCKER_CONFIG=/root/.docker/config.json
The autoscaler has this configuration
DRONE_AGENT_VOLUMES=/root/.docker:/root/.docker
I can check a working agent and see that the runner is working:
56af051cd495 drone/drone-runner-docker:1 "/bin/drone-runner-d…" 32 hours ago Up 32 hours 3000/tcp agent
I can do a docker exec -it /bin/sh into a working drone runner and see DRONE_DOCKER_CONFIG is set correctly and I can read from /root/.docker/config.json
When I use the CLI and type drone server info on a new agent I see:
Error: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
We have validated, today and several times, that the credentials we’re passing to docker works.
Our next step is to set DRONE_RUNNER_VOLUMES to /root/.docker, but we’re running out of options, and the rate limit keeps hitting us. Are we in some edge case with the autoscaler here?
An update on this.
After a lot of frustrated research, wondering why pipelines can pull private repos from docker hub but we’re still hitting a rate limit, we’ve narrowed down why we’re hitting the rate limit to the drone-docker plugin. Because we publish images to another registry (quay) we pull from docker hub and publish to quay - and it looks like drone-docker’s pulls are unauthenticated. We haven’t looked into a fix yet but one might be available. So all our dockerfiles with FROM python/blah for example are being pulled in an unauthenticated way.
We tested this by a multi-stage pipeline where we tried to trick drone into unauthenticated pulls of a private repo, all of which succeeded (meaning the drone step’s docker login worked, which is good) and a drone-docker step that built a container image with a FROM step pointing to a private repo the docker hub credentials we provided should have access to. That failed.
There’s still the lingering worry why we see the autoscaler was making unauthenticated docker hub pulls, even though the credential steps should be satisfied by our build. But maybe we’re doing something wrong there, hard to say.
I discuss this scenario in this comment, as well as possible solutions:
It turns out our fix with https://github.com/drone/drone-registry-plugin didn’t seem to work.
We have since started using /root/.docker/config.json as described in How to prevent DockerHub pull rate limit errors
We have ~55 pipelines with ~200 steps that build using plugin/docker - it’s untenable to simply modify all of those pipelines in a reasonable fashion, and we have no interest or time to build, deploy, and maintain a registry mirror.
Since we can’t modify pipelines or setup a mirror, we are trying to use DRONE_RUNNER_VOLUMES to have drone agents mount the host docker credentials into the pipelines step so that the daemon is already authenticated with dockerhub by the time it starts up. However we are hitting a problem where the variable is somehow defined twice in the agent’s configuration:
We are using a drone autoscaler to launch drone-docker-runner instances in AWS EC2, and providing some additional configuration by defining an agent configuration file using DRONE_AGENT_ENV_FILE.
The env file looks like this:
DRONE_SECRET_SECRET=<secret>
DRONE_SECRET_ENDPOINT=<snip>
DRONE_DOCKER_CONFIG=/root/.docker/config.json
DRONE_RUNNER_VOLUMES=/root/.docker:/root/.docker
All these environment variables are set as expected, but somehow DRONE_RUNNER_VOLUME ends up defined twice in the agent container that starts. The follow is the relevant portion of “docker inspect agent” for one of the auto-scaled agents. Since the empty version of DRONE_RUNNER_VOLUMES is defined second, the value we set is ignored, and the mount does not happen as expected.
"Env": [
"DRONE_SECRET_ENDPOINT=<snip>",
"DRONE_DOCKER_CONFIG=/root/.docker/config.json",
"DRONE_RUNNER_VOLUMES=/root/.docker:/root/.docker",
"DRONE_SECRET_SECRET=<snip>",
"DRONE_RPC_HOST=<snip>",
"DRONE_RPC_PROTO=https",
"DRONE_RPC_SERVER=<snip>",
"DRONE_RPC_SECRET=<snip>",
"DRONE_RUNNER_CAPACITY=2",
"DRONE_RUNNER_NAME=agent-unx6C5uD",
"DRONE_RUNNER_VOLUMES=",
"DRONE_RUNNER_DEVICES=",
"DRONE_RUNNER_PRIVILEGED_IMAGES=",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"GODEBUG=netdns=go",
"DRONE_PLATFORM_OS=linux",
"DRONE_PLATFORM_ARCH=amd64"
],
To be clear, we are a paying customer with a drone enterprise license, and so far the response to this issue has been lackluster. We need a solution, not a work around.
the DRONE_RUNNER_VOLUMES variable is used to configure global volumes that are mounted into each pipeline step, which is not going to be the solution you are looking for in this case. If you want to configure the autoscaler to create the runner (agent) container with a host volume mount to load the config.json file, you can use the following environment variable: https://autoscale.drone.io/reference/drone-agent-volumes/
I feel like we’re talking past one another. We have already configured the DRONE_AGENT_VOLUMES to allow the agent on each build machine to pull from dockerhub without additional configuration. This works as expected and has mostly solved the dockerhub rate limiting for us.
The problem we face now is that when using steps with ‘plugins/docker’ you can only provide a single set of credentials. We host our private images on quay, so we already provide those.
The options provided of setting up a repository mirror or using the setting.config for each pipeline step is simply not viable with our currently development capacity.
We hoped to use DRONE_RUNNER_VOLUMES to mount the agent’s docker credentials all pipeline steps, so that plugins/drone could pull images using an actual account. In local testing with ‘docker run’, this worked as intended, but we are currently blocked on the duplicate DRONE_RUNNER_VOLUMES as detailed above.
I’m not sure how that happened, and in some basic testing, I couldn’t get ‘docker run’ to reproduce that behavior locally. Unless I’m failing to see a typo in my env vars above, somehow the drone agent is starting with DRONE_RUNNER_VOLUMES defined twice.
the autoscaler accepts a named DRONE_RUNNER_VOLUMES variable which means you do not need to set using the freeform DRONE_AGENT_ENV_FILE. Using the former will ensure the value is correctly set, while using the latter would result in duplicate values.