We have a fairly heavy build/testing step and packaging step. We have autoscale on and noticed it will spin up a machine (AWS) for each pipeline when I have a diamond pattern with pipeline depends.
a starts with early exit tests
b and c both depend on a so we don’t spend tons of time if early exit fails.
d depends on b and c with failure notification or pushing archives if testing was good.
a / \ b c \ / d
If all pipelines were self-contained, this works great. b and c are run on different machines, so the heavy processes we want in parallel work as expected. My only thing to solve is publishing artifacts from b if both b and c are successful.
One option is pushing files from c into d somehow and then this automatically doesn’t happen if both b and c are successful.
Alternatively, if there is a way of depending on a step in pipeline b on pipeline c completing correctly.
A little more detailed diamond:
pre-check -> on failure pipeline / \ packaging testing (both depends_on pre-check) \ / (run in parallel with two machines) \ / push_packaging (on success) notify (on failure) (both depends_on packaging and testing)
The only thing I can figure out is some external temp file store that we push and pull to. Anyone have a nice solution to this?