It happens many times during the day but not always. Eventually I noticed that container pod showed all success logs but step in UI was still running and showed only half of logs. The step is running forever (hours) until you cancel build manually from UI.
We also tried new flags but still hanging
sounds like you probably stopped / shutdown / or restarted the runner while the pipeline was running. This is the only known root cause for an orphaned build.
Thanks for reply. For sure kube runner is not touched. There are 3 runner pods shown as healthy but for some reason step sometimes keeps running in UI (sequential execution) even it was completed successfully. Logs seem not having any warnings or errors related to an issue. One note I can add that there are more chances to hang if more than one job is running. Haven’t noticed that a single running job hangs. Do you have more ideas about the issue?