A quick insight on immutable delegates

Usually the delegate downloads a lot of runtime components during startup that includes startup scripts, delegate jars, 3rd party libraries and so on., posing security constraints to users. In such a scenario, 3rd party libraries with the security vulnerabilities can be downloaded without user’s attention and that was a major ASK from users to make them secure.

What does an immutable delegate mean?

  1. We cannot download anything in the delegate runtime. We had to move all the components that were supposed to be downloaded including 3rd party libraries like Kubectl, helm etc., and was bundled up in a delegate image in the delegate image build time. We do not have a watcher anymore.
  2. Manager and the delegate versions are decoupled and allowed to have Independent release candidates from the manager, using a ring deployment mechanism.
  3. Ephemeral pods for Kubernetes delegate, that is, you can easily replace any pod in a scenario, if you have moved a replica delegate and if the pod got stuck, they can be easily replaced.
  4. To support automatic upgrades, we have introduced a component called ‘upgrader’ and will be an optional component to manage the updates.

Benefits of immutable delegate:

  1. Users cannot scan delegate images
    Security scanning is currently not possible. If a user scans it, they need not have the same image in the production, when they install this delegate image.

  2. Users will not have control on upgrade
    Everytime we release our SaaS solution, the delegate gets upgraded automatically. So users have no control over it.

  3. Instability concerns
    Because of the current watcher-delegate mechanism especially in the startup time, delegate installation was not correct and we have constantly worked on it to address that area.

  4. Current delegates do not support cloud native best practices
    It’s our vision to make the delegate support, as our user’s follow them and install it on Kubernetes, that is, cloud native.

Common FAQs for immutable delegate:

  1. What type of delegates are supported by immutable delegates?
    This feature is only available for containerized delegates. We only support the Kubernetes delegate for now. Support for other containerized delegates will be available in coming weeks.

  2. What base image is used by the immutable delegates?
    Immutable delegate is based on redhat/ubi8-minimal:8.4

  3. Are delegate profiles supported by immutable delegates?
    Immutable delegates DO NOT support delegate profiles and are not available for NG. If the user wants to do something specific before the delegate starts then INIT_SCRIPT can be used.

  4. Is the image signed?
    Yes, the image is signed. You can run a command: docker trust inspect --pretty harness/delegate-immutable:22.03.74407

  5. Do I need to migrate all my existing delegates?
    You can migrate delegates over time. The existing(non-immutable delegate) can work with any new immutable delegates and need not be migrated.

  6. How to migrate my existing delegates to be immutable?
    They need to be re-installed(steps/workflows that use them need to be re-configured) and cannot be automated.

  7. Root vs Non-root
    Immutable delegates don’t have a root image. There are only two images for the same tag and both of them are non root,
    Out of the two, the first one has client tools like Kubectl, helm, chartmuseum while ‘minimal’ doesn’t have those tools. To have more tools in your image, you can build your own custom image.

  8. Does enabling the feature flag affect my existing delegate immutable?
    No, by enabling the feature flag, all your future delegates become immutable and don’t affect the existing delegates. You will need to migrate it to make it immutable.

  9. How can I use immutable delegates
    Immutable delegate image are different from existing delegate images you can now find them as harness/delegate-immutable in the dockerhub repo and in case if you haven’t configured an immutable image for your account yaml, then by default it will pick harness/delegate:latest (not an immutable image).


Immutable delegate is supported only as Kubernetes delegate on CG and NG. Inorder to generate appropriate harness-delegate.yml when installing a Kubernetes delegate, make sure you enable ‘USE_IMMUTABLE_DELEGATE’ feature flag for the account. Moreover, Harness Manager doesn’t really make much difference if you are installing [old] mutable or [new] immutable delegates and can work with both in parallel.