Pipeline fails with messages related to secrets in runners

Drone Support
#1

simple pipeline. kubernetes runner. setup as per online examples (role bindings, etc…).
making just a test thing. but won’t work…
making push from developer’s VM, all good, then it logs this:
{“level”:“debug”,“msg”:“api: read access granted”,“name”:“go_070921”,“namespace”:“devops”,“request-id”:“1xooLGZ85HojeFNjVdX078B2FQf”,“time”:“2021-09-07T16:05:44Z”,“user.login”:“devops”,“visibility”:“public”}

and runner logs this:

  • failed to create secret" error=“secrets is forbidden: User “system:serviceaccount:development:default” cannot create resource “secrets” in API group “” in the namespace “development”” namespace=development
  • “failed to delete secret” error=“secrets “drone-lcjzvvuk2ht4sr5w18xt” is forbidden: User “system:serviceaccount:development:default” cannot delete resource “secrets” in API group “” in the namespace “development”” namespace=development

drone UI says: go_app - clone: skipped and nothing more…

any ideas community please?

#2

same issue.


// https://github.com/drone/charts/tree/master/charts/drone-runner-kube
resource "helm_release" "drone-server" {
  name       = "drone-server"
  repository = "https://charts.drone.io"
  chart      = "drone"
  namespace  = var.namespace
  // TODO: version lock
  dynamic "set" {
    for_each = {
      "fullnameOverride"               = local.drone_svc_name
      "image.tag"                      = "2.4.0"
      "env.DRONE_SERVER_HOST"          = local.drone_server_uri
      "env.DRONE_SERVER_PROTO"         = "https"
      "env.DRONE_RPC_SECRET"           = var.drone_rpc_secret
      "env.DRONE_DATABASE_SECRET"      = var.drone_database_secret
      "env.DRONE_GITHUB_CLIENT_ID"     = var.drone_github_client_id
      "env.DRONE_GITHUB_CLIENT_SECRET" = var.drone_github_client_secret
      "env.DRONE_DATABASE_DRIVER"      = "postgres"
      "env.DRONE_DATABASE_DATASOURCE"  = var.rds_conn_string_drone
      "env.DRONE_S3_BUCKET"            = aws_s3_bucket.drone.id
      "env.DRONE_USER_FILTER"          = "myotherorg\\,myghorg"
      "env.DRONE_GIT_ALWAYS_AUTH"      = true
      "env.DRONE_TRACE"                = true
      "env.DRONE_DEBUG"                = true
      "AWS_ACCESS_KEY_ID"              = aws_iam_access_key.drone_server.id
      "AWS_SECRET_ACCESS_KEY"          = aws_iam_access_key.drone_server.secret
      "AWS_DEFAULT_REGION"             = data.aws_region.current.name
    }
    content {
      name  = set.key
      value = set.value
    }
  }
}

resource "helm_release" "drone-runner" {
  name       = "drone-runner"
  repository = "https://charts.drone.io"
  chart      = "drone-runner-kube"
  namespace  = var.namespace
  // TODO: version lock
  dynamic "set" {
    for_each = {
      "image.tag"                   = "1.0.0-rc.1"
      "env.DRONE_RPC_HOST"          = "${local.drone_svc_name}.${var.namespace}.svc.cluster.local"
      "env.DRONE_RPC_PROTO"         = "http"
      "env.DRONE_RPC_SECRET"        = var.drone_rpc_secret
      "env.DRONE_NAMESPACE_DEFAULT" = "tools"
      "env.DRONE_TRACE"             = true
      "env.DRONE_DEBUG"             = true
    }
    content {
      name  = set.key
      value = set.value
    }
  }
  values = [
    yamlencode({
      "rbac.buildNamespaces" = ["tools"]
    })
  ]
}```

trace msg=“secret: database: found matching secret” kind=secret name=slack_webhook thread=17

time=“2021-10-13T00:29:55Z” level=debug msg=“updated stage to running” build.id=16 build.number=8 repo.id=79 repo.name=lettuce repo.namespace=myorg2 stage.id=16 stage.name=default stage.number=1 thread=17

time=“2021-10-13T00:29:55Z” level=error msg=“failed to create secret” error=“secrets is forbidden: User “system:serviceaccount:tools:drone-runner-drone-runner-kube” cannot create resource “secrets” in API group “” in the namespace “tools”” namespace=tools pod=drone-cvvpk7kp4sshr03keoa2

time=“2021-10-13T00:29:56Z” level=debug msg=“destroying the pipeline environment” build.id=16 build.number=8 repo.id=79 repo.name=lettuce repo.namespace=imisti stage.id=16 stage.name=default stage.number=1 thread=17

time=“2021-10-13T00:30:01Z” level=error msg=“failed to delete secret” error=“secrets “drone-cvvpk7kp4sshr03keoa2” is forbidden: User “system:serviceaccount:tools:drone-runner-drone-runner-kube” cannot delete resource “secrets” in API group “” in the namespace “tools”” namespace=tools pod=drone-cvvpk7kp4sshr03keoa2

#3

cannot create resource “secrets” in API group “” in the namespace “tools”

The Kubernetes pipeline executes inside a Pod, and Drone secrets are injected into Pods as Kubernetes secrets that are created at runtime, using the Kubernetes API. This error indicates you have not given the Kubernetes runner sufficient permission to create a Kubernetes secret in the target namespace.

#4

Hi Everyone.

I also faced similar problem. Running a test echo example, blow is the logs from runner pod.

time="2022-02-25T19:12:28Z" level=error msg="failed to create secret" error="Post \"https://10.43.0.1:443/api/v1/namespaces/droneci/secrets\": dial tcp 10.43.0.1:443: i/o timeout" namespace=droneci pod=drone-1uxpkrn8265320eofrhi
time="2022-02-25T19:13:03Z" level=error msg="failed to delete secret" error="Delete \"https://10.43.0.1:443/api/v1/namespaces/droneci/secrets/drone-1uxpkrn8265320eofrhi\": dial tcp 10.43.0.1:443: i/o timeout" namespace=droneci pod=drone-1uxpkrn8265320eofrhi
time="2022-02-25T19:13:33Z" level=error msg="failed to delete pod" error="Delete \"https://10.43.0.1:443/api/v1/namespaces/droneci/pods/drone-1uxpkrn8265320eofrhi\": dial tcp 10.43.0.1:443: i/o timeout" namespace=droneci pod=drone-1uxpkrn8265320eofrhi

It seems like there is issue with rbac. but when i test the service account, it seems to be fine.

❯ kubectl auth can-i create secrets  -n=droneci --as=system:serviceaccount:droneci:drone-runner
yes

kubectl auth can-i create po  -n=droneci --as=system:serviceaccount:droneci:drone-runner
yes

I share my k8s yaml file for ref.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: droneci
  name: drone-runner
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - create
  - delete
- apiGroups:
  - ""
  resources:
  - pods
  - pods/log
  verbs:
  - get
  - create
  - delete
  - list
  - watch
  - update

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: drone-runner
  namespace: droneci
subjects:
- kind: ServiceAccount
  name: drone-runner
  namespace: droneci
roleRef:
  kind: Role
  name: drone-runner
  apiGroup: rbac.authorization.k8s.io

Runner Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: drone-runner
  namespace: droneci
  labels:
    app.kubernetes.io/name: drone-runner
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: drone
  template:
    metadata:
      labels:
        app.kubernetes.io/name: drone
    spec:
      serviceAccountName: drone-runner
      containers:
      - name: runner
        image: drone/drone-runner-kube:linux-amd64
        ports:
        - containerPort: 3000
        env:
        - name: DRONE_NAMESPACE_DEFAULT
          value: droneci
        - name: DRONE_SERVICE_ACCOUNT_DEFAULT
          value: drone-runner
        - name: DRONE_RPC_HOST
          value: droneserver.droneci.svc.cluster.local
        - name: DRONE_RPC_PROTO
          value: http
        - name: DRONE_RPC_SECRET 
          valueFrom:
            secretKeyRef:
              name: drone-server-secret
              key: DRONE_RPC_SECRET
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: drone-runner
  labels:
    app.kubernetes.io/name: drone-runner

This is my first time trying droneCI, I love the simplicity hope can make it work with my k8s cluster.
Thank You.