Extension raw pipeline config encoding

Hello,

We are hitting issues with drone pipeline serialization and how drone validates yaml key values with interpolated values and indicator tokens. This is an issue for conversion extensions in pipelines when serializing the raw yaml config of the pipeline received after modification.

The two issues we have found so far are:

  1. yaml line wrapping behavior for lines longer than 80 characters. This is likely a symptom of the yaml endcoding/decoding library but surfacing it as an issue more broadly working with yaml serialized data in extensions since this isn’t configurable in a number of libraries.
input:
 commands:
  - aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa b
  - another command

will be marshaled as:

 commands:
  - aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    b
  - another command

We can solve this by changing our yaml library but I want to mention it as an issue with using yaml as the raw format for extension pipeline data.

  1. values including drone variables for interpolation and yaml stop characters are invalid after unmarhsal → marshal

Example:
The following is valid yaml with the double quoted values string value.

- image: plugins/kaniko-ecr:latest-kaniko1.8.1
  name: build
  settings:
    no_push: true
    thing: "key.subkey.image=path.to.a.repohost.com/repo/name:${DRONE_TAG},key.subkey.envVars.PREFIX_WORKER_IMAGE=path.to.a.repohost.com/repo/name:${DRONE_TAG}"

If this is passed through a yaml unmarshal marshal cycle it yeilds:

- image: plugins/kaniko-ecr:latest-kaniko1.8.1
  name: build
  settings:
    no_push: true
    thing: key.subkey.image=path.to.a.repohost.com/repo/name:${DRONE_TAG},key.subkey.envVars.PREFIX_WORKER_IMAGE=path.to.a.repohost.com/repo/name:${DRONE_TAG}

The unquoted values value fails validation by the drone CLI.

You can replicate both of these behaviors with the drone 1.6.2 cli if you generate the yaml formatted. The generated yaml fails to line using the same drone cli. Adding escaped quotes to the values value in the starlark yields a single quoted string containing double quotes which would fail to interpolate.

Does drone provide a yaml module that we can use to safely decode and encode to avoid this issues? Is there a proper way to provide json serialized bytes back and forth to extension routers?

1 Like

A bit more detail on issue #2:

The flow of a .drone.yaml goes to conversion extensions first. Before conversion it still has the user specified quotes.

- image: plugins/kaniko-ecr:latest-kaniko1.8.1
  name: build
  settings:
    thing: "key.subkey.image=path.to.a.repohost.com/repo/name:${DRONE_TAG}"
  when:
    event:
    - tag

During conversion, a yaml unmarshal marshal cycle (with “gopkg.in/yaml.v3”), the quotes are stripped off because the lib determines there is a character $ after the : and quotes are unnecessary.

    thing: key.subkey.image=path.to.a.repohost.com/repo/name:${DRONE_TAG}

After conversion, I think it goes to Drone server, on a non-tag event ${DRONE_TAG} expands to nothing/whitespace, resulting in:

    thing: key.subkey.image=path.to.a.repohost.com/repo/name:

Which results in this error in Drone:

yaml: line 157: mapping values are not allowed in this context

This only occurs because name:${DRONE_TAG} is the last thing on the line. Validation passes if there is any non-whitespace character that follows.

Question for Drone team
There must be others who use conversion extensions and unmarshal/marshal yaml. And they must use Drone substitutions. How can we handle this?