K8s runner pipeline mounting external volumes

Hi all,

We have a setup of Drone deployed into an Azure AKS cluster using the K8S runner and we’d like to attach external volumes into the pipelines so we can offload the caching of build artifacts (i.e. dependencies) into an Azure FileShare.

We have already configured the appropriate persistent volume claims to allow Kubernetes to mount the Azure File Share as a volume into the pods that request it, so, for example, we can mount that storage into a /opt/drone/cache folder in the Drone Runner using extraVolumes and extraVolumeMounts.

Then we configure the pipelines to use the previously mentioned folder from the host using

kind: pipeline
type: kubernetes
name: default

steps:
  - name: build
    image: ...
    volumes:
      - name: cache
        path: /root/.ivy2 # This is where Scala's SBT stores the dependencies

volumes:
  - name: cache
    host:
       path: /opt/drone/cache

When the build kicks in and completes, we expect being able to see the cached artifacts in the Azure Storage, however we see the storage being empty. Which makes us think that Drone is not mounting the mentioned volume into the pipeline pod.

If we configure the extra volume mounts in the drone server deployment, notice the same thing.

How should we configure the Drone deployment so we can do that? Does this need to go in the runner policy instead or is not possible to do right now?

@alonsodomin,

Could you please enable trace logging and share the logs so that we can review and suggest accordingly.

Regards,
Harness support

@csgit Just enabled debug and tracing logs, this is what can be seen in the runner logs:

time="2021-04-15T13:04:24Z" level=debug msg="stage received" stage.id=9487 stage.name=validate stage.number=1 thread=4
time="2021-04-15T13:04:24Z" level=debug msg="stage received" stage.id=9487 stage.name=validate stage.number=1 thread=6
time="2021-04-15T13:04:24Z" level=debug msg="stage accepted" stage.id=9487 stage.name=validate stage.number=1 thread=4
time="2021-04-15T13:04:24Z" level=debug msg="stage accepted by another runner" stage.id=9487 stage.name=validate stage.number=1 thread=6
time="2021-04-15T13:04:24Z" level=debug msg="poller: request stage from remote server" thread=6
time="2021-04-15T13:04:24Z" level=debug msg="stage details fetched" build.id=6753 build.number=953 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 thread=4
time="2021-04-15T13:04:24Z" level=trace msg="secret: database: found matching secret" kind=secret name=bankifibot_ghtoken thread=4
time="2021-04-15T13:04:24Z" level=trace msg="secret: database: found matching secret" kind=secret name=artifactory_password thread=4
time="2021-04-15T13:04:24Z" level=trace msg="secret: database: found matching secret" kind=secret name=slack_webhook thread=4
time="2021-04-15T13:04:24Z" level=trace msg="secret: database: found matching secret" kind=secret name=dockerconfig thread=4
time="2021-04-15T13:04:24Z" level=debug msg="updated stage to running" build.id=6753 build.number=953 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 thread=4
time="2021-04-15T13:04:32Z" level=debug msg="received exit code 0" build.id=6753 build.number=953 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 step.name=clone thread=4
time="2021-04-15T13:04:36Z" level=debug msg="received exit code 0" build.id=6753 build.number=953 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 step.name="fetch tags" thread=4
time="2021-04-15T13:04:54Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:04:54Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:05:34Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:05:34Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:06:14Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:06:14Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:06:54Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:06:54Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:07:09Z" level=debug msg="stage received" stage.id=9488 stage.name=build stage.number=1 thread=7
time="2021-04-15T13:07:09Z" level=debug msg="stage received" stage.id=9488 stage.name=build stage.number=1 thread=8
time="2021-04-15T13:07:09Z" level=debug msg="stage accepted by another runner" stage.id=9488 stage.name=build stage.number=1 thread=8
time="2021-04-15T13:07:09Z" level=debug msg="poller: request stage from remote server" thread=8
time="2021-04-15T13:07:09Z" level=debug msg="stage accepted" stage.id=9488 stage.name=build stage.number=1 thread=7
time="2021-04-15T13:07:10Z" level=debug msg="stage details fetched" build.id=6754 build.number=85 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=artifactory_password thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=bankifibot_ghtoken thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=azenv_central_subscription_id thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=azenv_central_registry_username thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=azenv_central_registry_password thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=azenv_central_tenant_id thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=azure_client_id thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=azure_client_secret thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=slack_webhook thread=7
time="2021-04-15T13:07:10Z" level=trace msg="secret: database: found matching secret" kind=secret name=dockerconfig thread=7
time="2021-04-15T13:07:10Z" level=debug msg="updated stage to running" build.id=6754 build.number=85 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 thread=7
time="2021-04-15T13:07:20Z" level=debug msg="received exit code 0" build.id=6754 build.number=85 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 step.name=clone thread=7
time="2021-04-15T13:07:24Z" level=debug msg="received exit code 0" build.id=6754 build.number=85 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 step.name="fetch tags" thread=7
time="2021-04-15T13:07:34Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:07:39Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:07:40Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:08:06Z" level=debug msg="received exit code 1" build.id=6754 build.number=85 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 step.name="Build Pull Request" thread=7
time="2021-04-15T13:08:08Z" level=debug msg="received exit code 0" build.id=6754 build.number=85 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 step.name=notify thread=7
time="2021-04-15T13:08:08Z" level=debug msg="destroying the pipeline environment" build.id=6754 build.number=85 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 thread=7
time="2021-04-15T13:08:13Z" level=debug msg="successfully destroyed the pipeline environment" build.id=6754 build.number=85 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 thread=7
time="2021-04-15T13:08:13Z" level=debug msg="updated stage to complete" build.id=6754 build.number=85 duration=58 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 thread=7
time="2021-04-15T13:08:13Z" level=debug msg="poller: request stage from remote server" thread=7
time="2021-04-15T13:08:13Z" level=trace msg="http: context canceled"
time="2021-04-15T13:08:13Z" level=debug msg="done listening for cancellations" build.id=6754 build.number=85 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9488 stage.name=build stage.number=1 thread=7
time="2021-04-15T13:08:14Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:08:19Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:08:39Z" level=debug msg="stage received" stage.id=9489 stage.name=build stage.number=1 thread=9
time="2021-04-15T13:08:39Z" level=debug msg="stage received" stage.id=9489 stage.name=build stage.number=1 thread=10
time="2021-04-15T13:08:39Z" level=debug msg="stage received" stage.id=9489 stage.name=build stage.number=1 thread=11
time="2021-04-15T13:08:39Z" level=debug msg="stage accepted" stage.id=9489 stage.name=build stage.number=1 thread=9
time="2021-04-15T13:08:39Z" level=debug msg="stage accepted by another runner" stage.id=9489 stage.name=build stage.number=1 thread=11
time="2021-04-15T13:08:39Z" level=debug msg="poller: request stage from remote server" thread=11
time="2021-04-15T13:08:39Z" level=debug msg="stage details fetched" build.id=6755 build.number=86 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=artifactory_password thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=bankifibot_ghtoken thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=azenv_central_registry_username thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=azenv_central_registry_password thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=azenv_central_tenant_id thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=azure_client_secret thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=azenv_central_subscription_id thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=azure_client_id thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=slack_webhook thread=9
time="2021-04-15T13:08:39Z" level=trace msg="secret: database: found matching secret" kind=secret name=dockerconfig thread=9
time="2021-04-15T13:08:39Z" level=debug msg="stage accepted by another runner" stage.id=9489 stage.name=build stage.number=1 thread=10
time="2021-04-15T13:08:39Z" level=debug msg="poller: request stage from remote server" thread=10
time="2021-04-15T13:08:39Z" level=debug msg="updated stage to running" build.id=6755 build.number=86 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 thread=9
time="2021-04-15T13:08:54Z" level=debug msg="received exit code 0" build.id=6755 build.number=86 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 step.name=clone thread=9
time="2021-04-15T13:08:54Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:09:00Z" level=debug msg="received exit code 0" build.id=6755 build.number=86 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 step.name="fetch tags" thread=9
time="2021-04-15T13:09:09Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:09:09Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:09:34Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:09:39Z" level=debug msg="received exit code 1" build.id=6755 build.number=86 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 step.name="Build Pull Request" thread=9
time="2021-04-15T13:09:41Z" level=debug msg="received exit code 0" build.id=6755 build.number=86 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 step.name=notify thread=9
time="2021-04-15T13:09:41Z" level=debug msg="destroying the pipeline environment" build.id=6755 build.number=86 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 thread=9
time="2021-04-15T13:09:46Z" level=debug msg="successfully destroyed the pipeline environment" build.id=6755 build.number=86 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 thread=9
time="2021-04-15T13:09:46Z" level=debug msg="updated stage to complete" build.id=6755 build.number=86 duration=62 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 thread=9
time="2021-04-15T13:09:46Z" level=debug msg="poller: request stage from remote server" thread=9
time="2021-04-15T13:09:46Z" level=trace msg="http: context canceled"
time="2021-04-15T13:09:46Z" level=debug msg="done listening for cancellations" build.id=6755 build.number=86 repo.id=98 repo.name=proxy-notification-ms repo.namespace=BankiFi stage.id=9489 stage.name=build stage.number=1 thread=9
time="2021-04-15T13:09:49Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:10:09Z" level=debug msg="received exit code 1" build.id=6753 build.number=953 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 step.name=validate thread=4
time="2021-04-15T13:10:11Z" level=debug msg="received exit code 0" build.id=6753 build.number=953 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 step.name=notify thread=4
time="2021-04-15T13:10:12Z" level=debug msg="destroying the pipeline environment" build.id=6753 build.number=953 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 thread=4
time="2021-04-15T13:10:14Z" level=trace msg="http: no content returned: re-connect and re-try"
time="2021-04-15T13:10:17Z" level=debug msg="successfully destroyed the pipeline environment" build.id=6753 build.number=953 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 thread=4
time="2021-04-15T13:10:17Z" level=debug msg="updated stage to complete" build.id=6753 build.number=953 duration=347 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 thread=4
time="2021-04-15T13:10:17Z" level=debug msg="poller: request stage from remote server" thread=4
time="2021-04-15T13:10:24Z" level=trace msg="http: context canceled"
time="2021-04-15T13:10:24Z" level=debug msg="done listening for cancellations" build.id=6753 build.number=953 repo.id=16 repo.name=platform repo.namespace=BankiFi stage.id=9487 stage.name=validate stage.number=1 thread=4
time="2021-04-15T13:10:29Z" level=trace msg="http: no content returned: re-connect and re-try"

For sake of completion, this is the output from kubectl when describing the runner pod:

Name:         default-drone-runner-kube-6b76d5f556-6xgv6
Namespace:    drone
Priority:     0
Node:         aks-overspill-73472730-vmss000008/10.1.1.58
Start Time:   Thu, 15 Apr 2021 14:02:50 +0100
Labels:       app.kubernetes.io/component=drone-runner-kube
              app.kubernetes.io/instance=default
              app.kubernetes.io/name=drone-runner-kube
              pod-template-hash=6b76d5f556
Annotations:  checksum/secrets: bad092f483a39ab149c361b3705111e229d7bd973055d38213a9b7fb3dcdcafa
Status:       Running
IP:           10.1.1.66
IPs:
  IP:           10.1.1.66
Controlled By:  ReplicaSet/default-drone-runner-kube-6b76d5f556
Containers:
  server:
    Container ID:   containerd://76fe79f57dbf83538ae9a4c2a17ddcb8cdaaca39a71dc392bfc7e8dd8006cb4d
    Image:          drone/drone-runner-kube:1.0.0-beta.6
    Image ID:       docker.io/drone/[email protected]:170aa6766441df9e76a44b529dedeb5ac84a2edb81b7bc0a527cbbfabf8c35f2
    Port:           3000/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 15 Apr 2021 14:02:52 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     3
      memory:  500Mi
    Requests:
      cpu:     2
      memory:  250Mi
    Environment Variables from:
      default-drone-runner-kube  ConfigMap  Optional: false
      drone-runner-config        Secret     Optional: false
    Environment:                 <none>
    Mounts:
      /etc/runner-config from config-files (ro)
      /opt/cache from cache (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-drone-runner-kube-token-49rcj (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  config-files:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      drone-runner-configfiles
    Optional:  false
  cache:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  cache
    ReadOnly:   false
  default-drone-runner-kube-token-49rcj:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-drone-runner-kube-token-49rcj
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

The yaml configuration you posted for mounting volumes looks correct to me. For reference, you can see a sample yaml configuration that we use in our integration tests here [1]. I feel like you could definitively test whether or not drone is mounting the volume with the below pipeline:

kind: pipeline
type: kubernetes
name: default

steps:
  - name: write
    image: alpine
    commands:
      - echo "hello" > /root/.ivy2/greetings.txt
    volumes:
      - name: cache
        path: /root/.ivy2

  - name: read
    image: alpine
    commands:
      - ls -la /root/.ivy2
      - cat /root/.ivy2/greetings.txt
    volumes:
      - name: cache
        path: /root/.ivy2

volumes:
  - name: cache
    host:
       path: /opt/drone/cache

However, please note that the above mounts a host volume, not a persistent volume claim. I believe configuring a persistent volume claim requires different syntax than mounting a host volume.

[1] https://github.com/drone-runners/drone-runner-kube/blob/master/samples/volume_host.yml

If we configure the extra volume mounts in the drone server deployment, notice the same thing.

I’m not sure I fully understand, since the Drone server does not have any configuration options that relate to kubernetes or volume mounting.

@bradrydzewski thanks for checking that in your side.

Regarding your comment:

I’m not sure I fully understand, since the Drone server does not have any configuration options that relate to kubernetes or volume mounting.

The chart in here seems to have the options for extraVolumes and extraVolumeMounts. Regardless, we used those options but the drone runner instead: charts/values.yaml at master · drone/charts · GitHub

We may have misunderstood how to take advantage of those settings. Our understanding was that we could mount a Persistent Volume Claim into the Drone Runner (using extraVolumes and extraVolumeMounts) and then in the pipeline, as you showed before, we could configure the pipeline to mount a host volume that just matches the same folder as where there PVC has been mounted in the Runner. The assumption is that when a new pipeline gets triggered, the host volume mounted from Drone will happen to be the PVC previously mounted.

It seems that our assumption is clearly not how the runner works and wondering if such a thing could be done. It would be very beneficial for us being able to share the caches across several pipelines running in different nodes as this kind of caches can get very big over time and being able to share them can save a lot of time out of our builds.

Yes it works a little differently. The kubernetes runner launches pipelines as pods on your clusters using the kubernetes API. This means mounting volumes into the runner’s Pod will have no impact on the pipeline Pods that the runner schedules on your cluster, since they are separate.

We merged a pull request that allows mounting a persistent volume claim that you may want to explore https://github.com/drone-runners/drone-runner-kube/pull/28.

Thanks for that tip, we just had the chance to give it a go right now and it does provide what we were expecting.

Many thanks again, worth adding that to the public documentation.