[Kubernetes Deployment] Rollout deployment error: Invalid value: "": may not be specified when `value` is not empty

Hello everyone :vulcan_salute:
Harness Logo

Introduction

We have no doubt that Harness is the best CD platform in the universe and makes our deployment life easier for Kubernetes environments. But we also need to be prepared for the pranks that the Kubernetes play on us.

Let’s see how Harness can behave when one of these Kubernetes tricks impacts our deployment. Here’s the error we see in our deployment logs, which we’ll cover in this article

ERROR   2021-09-17 17:04:03    Error from server (Invalid): error when applying patch:
ERROR   2021-09-17 17:04:03    {"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"app.kubernetes.io/managed-by\":\"Helm\",\"kubernetes.io/change-cause\":\"kubectl apply --kubeconfig=config --filename=manifests.yaml --record=true\"},\"labels\":{\"app-id\":\"secondary\"},\"name\":\"new-app-harness-example-deployment\",\"namespace\":\"pedro-staging\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"new-app-harness-example\"}},\"template\":{\"metadata\":{\"labels\":{\"app\":\"new-app-harness-example\",\"harness.io/release-name\":\"release-f3fe376d-0d43-3346-a22b-5be4ba45247c\"}},\"spec\":{\"containers\":[{\"env\":[{\"name\":\"SECRET_USERNAME\",\"valueFrom\":{\"secretKeyRef\":{\"key\":\"username\",\"name\":\"mysecret1-2\"}}},{\"name\":\"SECRET_PASSWORD\",\"valueFrom\":{\"secretKeyRef\":{\"key\":\"password\",\"name\":\"mysecret1-2\"}}},{\"name\":\"SECRET_PASSWORD_2\",\"value\":\"0.0.7\"}],\"image\":\"redis\",\"name\":\"new-app-harness-example\"}]}}}}\n"}},"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"new-app-harness-example"}],"containers":[{"$setElementOrder/env":[{"name":"SECRET_USERNAME"},{"name":"SECRET_PASSWORD"},{"name":"SECRET_PASSWORD_2"}],"env":[{"name":"SECRET_USERNAME","valueFrom":{"secretKeyRef":{"name":"mysecret1-2"}}},{"name":"SECRET_PASSWORD","valueFrom":{"secretKeyRef":{"key":"password","name":"mysecret1-2"}}}],"name":"new-app-harness-example"}]}}}}
ERROR   2021-09-17 17:04:03    to:
ERROR   2021-09-17 17:04:03    Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
ERROR   2021-09-17 17:04:03    Name: "new-app-harness-example-deployment", Namespace: "pedro-staging"
ERROR   2021-09-17 17:04:03    Object: &{map["kind":"Deployment" "apiVersion":"apps/v1" "metadata":map["managedFields":[map["operation":"Update" "apiVersion":"apps/v1" "time":"2021-09-17T19:49:18Z" "fieldsType":"FieldsV1" "fieldsV1":map["f:metadata":map["f:annotations":map["f:kubectl.kubernetes.io/last-applied-configuration":map[] "f:kubernetes.io/change-cause":map[] ".":map[] "f:app.kubernetes.io/managed-by":map[]] "f:labels":map[".":map[] "f:app-id":map[]]] "f:spec":map["f:revisionHistoryLimit":map[] "f:selector":map[] "f:strategy":map["f:rollingUpdate":map[".":map[] "f:maxSurge":map[] "f:maxUnavailable":map[]] "f:type":map[]] "f:template":map["f:metadata":map["f:labels":map[".":map[] "f:app":map[] "f:harness.io/release-name":map[]]] "f:spec":map["f:containers":map["k:{\"name\":\"new-app-harness-example\"}":map[".":map[] "f:env":map[".":map[] "k:{\"name\":\"SECRET_PASSWORD\"}":map[".":map[] "f:name":map[]] "k:{\"name\":\"SECRET_PASSWORD_2\"}":map[".":map[] "f:name":map[] "f:value":map[]] "k:{\"name\":\"SECRET_USERNAME\"}":map["f:name":map[] "f:valueFrom":map["f:secretKeyRef":map[".":map[] "f:key":map[] "f:name":map[]] ".":map[]] ".":map[]]] "f:image":map[] "f:imagePullPolicy":map[] "f:name":map[] "f:resources":map[] "f:terminationMessagePath":map[] "f:terminationMessagePolicy":map[]]] "f:dnsPolicy":map[] "f:restartPolicy":map[] "f:schedulerName":map[] "f:securityContext":map[] "f:terminationGracePeriodSeconds":map[]]] "f:progressDeadlineSeconds":map[] "f:replicas":map[]]] "manager":"kubectl"] map["fieldsV1":map["f:spec":map["f:template":map["f:spec":map["f:containers":map["k:{\"name\":\"new-app-harness-example\"}":map["f:env":map["k:{\"name\":\"SECRET_PASSWORD\"}":map["f:value":map[]]]]]]]]] "manager":"kubectl-edit" "operation":"Update" "apiVersion":"apps/v1" "time":"2021-09-17T20:03:46Z" "fieldsType":"FieldsV1"] map["fieldsV1":map["f:metadata":map["f:annotations":map["f:deployment.kubernetes.io/revision":map[]]] "f:status":map["f:conditions":map[".":map[] "k:{\"type\":\"Available\"}":map["f:lastUpdateTime":map[] "f:message":map[] "f:reason":map[] "f:status":map[] "f:type":map[] ".":map[] "f:lastTransitionTime":map[]] "k:{\"type\":\"Progressing\"}":map["f:lastUpdateTime":map[] "f:message":map[] "f:reason":map[] "f:status":map[] "f:type":map[] ".":map[] "f:lastTransitionTime":map[]]] "f:observedGeneration":map[] "f:readyReplicas":map[] "f:replicas":map[] "f:updatedReplicas":map[] "f:availableReplicas":map[]]] "manager":"kube-controller-manager" "operation":"Update" "apiVersion":"apps/v1" "time":"2021-09-17T20:03:47Z" "fieldsType":"FieldsV1"]] "namespace":"pedro-staging" "resourceVersion":"29039258" "generation":'\x02' "creationTimestamp":"2021-09-17T19:49:18Z" "labels":map["app-id":"secondary"] "annotations":map["kubernetes.io/change-cause":"kubectl apply --kubeconfig=config --filename=manifests.yaml --record=true" "app.kubernetes.io/managed-by":"Helm" "deployment.kubernetes.io/revision":"2" "kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"app.kubernetes.io/managed-by\":\"Helm\",\"kubernetes.io/change-cause\":\"kubectl apply --kubeconfig=config --filename=manifests.yaml --record=true\"},\"labels\":{\"app-id\":\"secondary\"},\"name\":\"new-app-harness-example-deployment\",\"namespace\":\"pedro-staging\"},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"app\":\"new-app-harness-example\"}},\"template\":{\"metadata\":{\"labels\":{\"app\":\"new-app-harness-example\",\"harness.io/release-name\":\"release-f3fe376d-0d43-3346-a22b-5be4ba45247c\"}},\"spec\":{\"containers\":[{\"env\":[{\"name\":\"SECRET_USERNAME\",\"valueFrom\":{\"secretKeyRef\":{\"key\":\"username\",\"name\":\"mysecret1-1\"}}},{\"name\":\"SECRET_PASSWORD\",\"valueFrom\":{\"secretKeyRef\":{\"key\":\"password\",\"name\":\"mysecret1-1\"}}},{\"name\":\"SECRET_PASSWORD_2\",\"value\":\"0.0.7\"}],\"image\":\"redis\",\"name\":\"new-app-harness-example\"}]}}}}\n"] "name":"new-app-harness-example-deployment" "uid":"94143aa2-d668-4055-a3df-cf3db70fa13c"] "spec":map["revisionHistoryLimit":'\n' "progressDeadlineSeconds":'\u0258' "replicas":'\x01' "selector":map["matchLabels":map["app":"new-app-harness-example"]] "template":map["spec":map["containers":[map["terminationMessagePath":"/dev/termination-log" "terminationMessagePolicy":"File" "imagePullPolicy":"Always" "name":"new-app-harness-example" "image":"redis" "env":[map["name":"SECRET_USERNAME" "valueFrom":map["secretKeyRef":map["name":"mysecret1-1" "key":"username"]]] map["name":"SECRET_PASSWORD" "value":"pass"] map["name":"SECRET_PASSWORD_2" "value":"0.0.7"]] "resources":map[]]] "restartPolicy":"Always" "terminationGracePeriodSeconds":'\x1e' "dnsPolicy":"ClusterFirst" "securityContext":map[] "schedulerName":"default-scheduler"] "metadata":map["creationTimestamp":<nil> "labels":map["app":"new-app-harness-example" "harness.io/release-name":"release-f3fe376d-0d43-3346-a22b-5be4ba45247c"]]] "strategy":map["rollingUpdate":map["maxUnavailable":"25%" "maxSurge":"25%"] "type":"RollingUpdate"]] "status":map["updatedReplicas":'\x01' "readyReplicas":'\x01' "availableReplicas":'\x01' "conditions":[map["lastUpdateTime":"2021-09-17T19:49:19Z" "lastTransitionTime":"2021-09-17T19:49:19Z" "reason":"MinimumReplicasAvailable" "message":"Deployment has minimum availability." "type":"Available" "status":"True"] map["lastTransitionTime":"2021-09-17T19:49:18Z" "reason":"NewReplicaSetAvailable" "message":"ReplicaSet \"new-app-harness-example-deployment-67c8fbfcb9\" has successfully progressed." "type":"Progressing" "status":"True" "lastUpdateTime":"2021-09-17T20:03:48Z"]] "observedGeneration":'\x02' "replicas":'\x01']]}
ERROR   2021-09-17 17:04:03    for: "manifests.yaml": Deployment.apps "new-app-harness-example-deployment" is invalid: spec.template.spec.containers[0].env[1].valueFrom: Invalid value: "": may not be specified when `value` is not empty

Tutorial

Requirements

For this example, we will only need a basic Rolling deployment Workflow and some Deployment in our manifest that have environment variables populated by reference (to a secret, config map, among others), in this example I will populate them from a secret. For a better visibility, I’ve created a repository on GitHub that basically contains the resources existing in my Service Manifest

First Step

Let’s create our Service! For our newly created service we will use as a manifest all the resources that are in the repository I shared above:

Second Step

Let’s create an environment where our service will be deployed, there’s no secret here, basically, you define your target deployment infrastructure using a Harness Environment. Environments represent your deployment infrastructures, such as Dev, QA, Stage, Production, etc.

Environment Overview

Third Step

Finally, let’s now create our Rolling Workflow that will deploy our amazing service.

Workflow Overview


Now that everything is ready, we just need to hit the Deploy button for our Workflow for the first time :rocket:

Successfully deployed :tada:

But after our magnificent deployment process, for some reason one of the developers identified a problem with the secret value used in our ​​defined variables for our pods and manually edited the Deployments directly on the cluster using:

kubectl edit Deployment <deployment-name>

changing the secret referent to plain text in all our Deployments. Basically, updating from this:

      containers:
        - name: new-app-harness-example
          image: redis
          env:
            - name: SECRET_USERNAME
              valueFrom:
                secretKeyRef:
                  name: mysecret1
                  key: username
            - name: SECRET_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysecret1
                  key: password

to this:

      containers:
        - name: new-app-harness-example
          image: redis
          env:
            - name: SECRET_USERNAME
              valueFrom:
                secretKeyRef:
                  name: mysecret1
                  key: username
            - name: SECRET_PASSWORD
              value: 'pass'

After that, the secret value was fixed and we trigger our workflow again to update it properly and our deployments will again use the reference to our secret instead of a plain text value. easy-peasy

Ohh no :scream: , my Workflow is no longer working, I’m getting this weird error message…

Final Step

Now we see this gross error for any new deployment, so to solve it we need to again edit the resources that we changed manually and remove the modified entries in the process, and after that perform the Deploy of our Workflow again so that the resources are recreated correctly:

So let’s edit the deployment that is having issues manually in the cluster, removing all the manually modified entries with:

kubectl edit Deployment <deployment-name>

For example, updating from this:

    containers:
        - name: new-app-harness-example
          image: redis
          env:
            - name: SECRET_USERNAME
              valueFrom:
                secretKeyRef:
                  name: mysecret1
                  key: username
            - name: SECRET_PASSWORD
              value: 'pass'

to this:

containers:
        - name: new-app-harness-example
          image: redis
          env:
            - name: SECRET_USERNAME
              valueFrom:
                secretKeyRef:
                  name: mysecret1
                  key: username


Now that we have removed the entry that was changed and was causing the kubectl apply process to fail, let’s run our workflow again so that the variable excluded in the process is restored to our Deployment, but correctly.

All green again :tada:

Outcome

The error happens whenever a resource is edited directly and manually in the cluster after its default creation, and a new update on the manually edited resource is done through the standard process, kubectl apply for example So any manual changes made to Kubernetes resources deployed by the Harness process can present similar issues when a new deployment occurs, to avoid this, the ideal is not to perform any manual adjustments to the resources that are deployed through the Workflow process, but apply any change following the processes of release.


Now you don’t have to worry anymore about this trick that Kubernetes can present and continue exploring more possibilities with Harness and Kubernetes :rocket:

2 Likes