Error with autoscaled drone agents

We have an issue with agents spinning up and reporting an error.

Via the autoscaler CLI commands I can see

‘error during connect: Post “https://{IPADDRESS}:2376/v1.40/images/create?fromImage=drone%2Fdrone-runner-docker&tag=1”: EOF’

I can ssh into the agents, and can’t really see anything wrong with them, although I’m not sure what I’m meant to look for. Docker does seem to be running on them.

It’s been a while since I touched the setup, but we host our agents in AWS, and they have a custom cloudinit that loads in docker hub credentials so they don’t die to rate limiting. Pretty sure it’s the template from the docs with a config file jammed in.

It’d be nice to fix this because those agents wait around for a bit, preventing us from scaling to our maximum, until the autoscaler kills them (we’ve enabled the autoscaler reaper feature).