Drone docker runner on kubernetes

Drone Support
#7

@ashwilliams1 when running runner-kube on EKS + worker node, when I use on drone.yml vol

kind: pipeline
name: default
type: kubernetes

platform:
os: linux
arch: amd64

steps:

volumes:

  • name: dockersock
    host:
    path: /var/run/

if docker fails for some reason it carshes the Docker deamon on Kubernetes worker node!!

#8

and also change the permissions of the docker.sock and casuse the docker deamon on the host to be unavaialble

#9

this is from drone log
Login Succeeded
46 + docker pull registry.strln.net/thor/alpine_hardened:latest
47 Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

#10

this is one way I able to skip mount /var/run/docker.sock of the host machine

kind: pipeline
name: default
type: kubernetes

platform:
os: linux
arch: amd64

steps:

volumes:

- name: dockersock

path: /var/run/

when:
event:
- tag

services:

  • name: docker
    image: docker:dind
    privileged: true
    volumes:
    • name: dockersock
      path: /var/run

volumes:

  • name: dockersock
    temp: {}
#11

One thing that jumps out is I see you are mounting the docker socket from the host into the plugins/ecr step, however, this plugin uses docker-in-docker and starts its own docker daemon. Mounting a docker socket into this plugin, when the plugin tries to start its own docker-in-docker daemon and create its own docker socket, could be causing problems.

Have you tried using plugins/ecr without mounting the host machine docker socket?

if docker fails for some reason it crashes the Docker deamon on Kubernetes worker node!!

If running standard docker causes the host machine Docker daemon to crash on your Kubernetes node, you may consider opening an issue with the Docker project. This sounds like it would be a severe Docker bug.

#12

While we generally recommend against mounting the host machine docker socket when using the docker plugin (since the plugin is meant to use docker-in-docker) you can disable the docker-in-docker daemon with the following settings:

image: plugins/ecr
settings:
  daemon_off: true
  purge: true

This is important, because if you do not disable the docker-in-docker daemon it may overwrite the host machine docker socket, or may cause the plugin to fail (this could even be the reason your Docker daemon on the node is crashing, because its socket is being overwritten). Also the purge setting prevents the plugin from running docker prune on the host and deleting your host machine docker cache.

However, I would recommend using this plugin without mounting the host machine docker socket if possible. Mounting the host machine docker socket effectively grants your pipelines root access to the host machine, which may be undesirable from a security perspective.

#13

@bradrydzewski, @ashwilliams1 this is SUPER important information which clarify the symptom I had on my system and worth to publish it for anyone have this issue.
I mount the /var/run/docker.sock from the first place becasue when I building the Image on the Dockerfile the from is from private registry and I am getting the following error:

  • /usr/local/bin/docker build --rm=true -f ./Dockerfile -t 83a273e4ce9fbc0aaa673428f86063f45c01ecc9 . --pull=true --label org.label-schema.schema-version=1.0 --label org.label-schema.build-date=2020-06-29T19:55:20Z --label org.label-schema.vcs-ref=83a273e4ce9fbc0aaa673428f86063f45c01ecc9 --label org.label-schema.vcs-url=https://github.office.opendns.com/tlv-opsys/ops-deployer.git
    83 Sending build context to Docker daemon 315.9kB
    84 Step 1/25 : FROM registry.strln.net/thor/alpine_hardened:latest
    85 Get https://registry.strln.net/v2/thor/alpine_hardened/manifests/latest: unauthorized: authentication required
    86 time=“2020-06-29T19:55:20Z” level=fatal msg=“exit status 1”
    87

how can I tell drone/kubernetes to use secrets to pull the image from private registry both on Pipeline step and when building an image, this is critical and if I solved it I can skip mounting /var/run/docker.sock

#14

@ihakimi the docker daemon uses auth credentials to push and pull images. However, one thing I noticed is that you have two different registries (991726456769.dkr.ecr.us-east-1.amazonaws.com and registry.strln.net). Do they have different authentication credentials? Or can they use the same credentials? Are both of these ECR registries? Do they both require generated (short lived) credentials?

#15

Yes, for 991726456769.dkr.ecr.us-east-1.amazonaws.com](http://991726456769.dkr.ecr.us-east-1.amazonaws.com Kubernetes can pull it automatically with IAM roles to this ecr,
the problem is on http://registry.strln.net/ which uses temporary credentials and the login is using with CEC user+2MFA, and when do cut to dockerjson its hidden on mac chain so it quite complicated and not understand how to get this credential because of that I used this step before:

  • name: pull
    pull: default
    image: alpine
    commands:
#16

when do cut to dockerjson its hidden on mac chain so it quite complicated and not understand how to get this credential because of that I used this step before:

I have a mac and have run into this issue. To get the credentials I usually run docker login inside a container on my mac and login and then copy the credentials:

$ docker run -t -i docker /bin/sh
# docker login
# cat ~/.docker/config.json

You can actually store the entire contents of the config.json file as a secret, and then you can use this config file in your plugin, like this:

image: plugins/ecr
settings:
  config:
    from_secret: ...

Perhaps the combination of using the config.json for strln.net and IAM for 991726456769.dkr.ecr.us-east-1.amazonaws.com would solve the issue?

#17

@bradrydzewski BINGO
image: plugins/ecr
settings:
config:
from_secret: auth

Step 1/25 : FROM registry.strln.net/thor/alpine_hardened:latest
85 latest: Pulling from thor/alpine_hardened
86

the only probelm is this is short live credentails, and this not understand how to solve, I can do step before login but how can I pass this credentail to next step for example config: from_file: XZZ

#18

@ihakimi I feel like there are a few different options that could be employed, each with different pros and cons. One option would be to mount the .docker home path as a temporary volume, run docker login to generate the config.json file which would be saved to this volume, making it available to subsequent steps:

steps:
- name: login
  image: docker
  commands:
  - curl https://downloads.strln.net/install | bash
  - sl login
  - $(sl container registry auth generate)
  volumes:
  - name: dockerconfig
    path: /root/.docker

- name: build
  image: plugins/ecr
  settings:
    ...
  volumes:
  - name: dockerconfig
    path: /root/.docker

volumes:
- name: dockerconfig
  temp: {}

Another option could be to provide these credentials automatically using a custom extension. You would have to write the custom extension (we provide starter templates to help simplify this process). Another option would be to fork the ecr plugin and add some extra parameters and logic to generate the credentials for registry.strln.net (I would probably consider this option). Happy to discuss these other options in greater depth if you are intersted.

#19

@bradrydzewski thank you very much! the following pipeline work!!!
my question can I take the volume I create and leverage it for pull private images from steps
for example
- name: login
image: docker
commands:
- apk update
- apk add curl bash
- curl https://downloads.strln.net/install | bash
- sl login
- $(sl container registry auth generate)
volumes:
- name: dockerconfig
path: /root/.docker

- name: private_image
  image: registry.strln.net/XXX/alpine_hardened:latest

image_pull_secrets:
  from_secret_file: /root/.docker
  volumes:
    - name: dockerconfig
      path: /root/.docker

volumes:
- name: dockerconfig
  temp: {}

does it possible to get the image_pull_secret from file rather then secrets because of short live session credentials.

#20

@ihakimi Drone can pull pipeline images (defined by the image: attribute) using a static username and password, but it cannot generate temporary usernames and passwords (for example, like what you are doing with sl container registry auth generate). Unfortunately there is no standard for generating temporary credentials, and every registry implements its own custom interface, which means there is no way for Drone to support this feature in a generic or universal manner.

So to solve this problem we introduced registry extensions:
https://docs.drone.io/extensions/registry/
https://docs.drone.io/extensions/registry/#starter-project

You can create a registry extension that generates the credentials (by interface with your private registry) and return the credentials back to Drone. This is going to be your best option because it will simplify everyone’s pipeline since it will automatically handle credential configuration. Extensions are simple REST microservices and we even provide starter projects that you can use to accelerate development.

#21

Hi @bradrydzewski,

I am starting to build the service using the starter-project, I using https://github.com/drone/drone-go/blob/master/plugin/registry/handler.go and created the secret using
openssl rand -hex 16
and start the service but don’t know what to pass on the curl command in order to test it.
15:07 $ curl -H “Authorization: basic XXXXXX” http://localhost:3000
Invalid or Missing Signature
and tried different authorization but not work

#22

@ihakimi the request needs to be signed using http-signatures which can make testing with curl difficult. Instead you can use the drone CLI to test the extension. I believe the following command should work:

$ export DRONE_REGISTRY_ENDPOINT=http://...
$ export DRONE_REGISTRY_SECRET=...
$ drone plugins registry list

When you write an extension you can use repository and build metadata to dynamically determine the response. For example, you may want to use the repository name or build event type to determine the response. This data can also be simulated from the command line using the command line flags:

   --ref value              git reference (default: "refs/heads/master")
   --source value           source branch
   --target value           target branch
   --before value           commit sha before the change
   --after value            commit sha after the change
   --event value            build event
   --repo value             repository name
#23

thanks @bradrydzewski got response from service, now need to add my dynamic credentials in order to test it

#24

Hi @bradrydzewski,
I built a sevice for image priave registry puller
If I put private image on step its workign but when using the ecr/plugin and on the Dockerfile there is private image url its fails

Step 1/25 : FROM registry.strln.net/thor/alpine_hardened:latest
85 Get https://registry.strln.net/v2/thor/alpine_hardened/manifests/latest: unauthorized: authentication required
86 time=“2020-07-02T13:16:18Z” level=fatal msg=“exit status 1”
87

steps:
- name: login1
  image: registry.strln.net/thor/alpine_hardened:latest
  pull: always
  commands:
    - apk update

- name: push-to-ecr-barnch
  image: plugins/ecr
  settings:
    create_repository: true
    dockerfile: ./Dockerfile
    region: us-east-1
    registry: 991726456769.dkr.ecr.us-east-1.amazonaws.com
    repo: 991726456769.dkr.ecr.us-east-1.amazonaws.com/opsys/ops-deployer
    tags:
    - "${DRONE_BRANCH}"
  environment:
    PLUGIN_PULL_IMAGE: true
    PLUGIN_REPOSITORY_POLICY: ecr_repository_policy.json
  when:
    event:
    - push
#25

@ihakimi the registry credential plugin provides the runner with credentials to pull pipeline step images, however, these credentials are not shared with plugins for security reasons. But no worries, we have another extension type that can be used to provide plugins with default configuration parameters (called an environment extension). We can help you combine these into a single Go program so that you can have a single codebase and share code. How about we setup a quick 30 minute call to walk through this?

#26

sure, if we can have it today later on in about 3 hours it will be great, you have my info