Drone-docker plugin insecure registry

I have drone CI pipeline wired up to GitLab, and the resulting container image won’t push to a local insecure registry.

I’m running docker on CentOS 7.2 w/ OverlayFS configured and insecure registry configured.

Firewalld and SELinux are running. I’ve opened these ports:

[[email protected] nodejs-helloworld]$ sudo firewall-cmd --list-ports
25432/tcp 8000/tcp 16379/tcp 15432/tcp 10080/tcp 10022/tcp
[[email protected] nodejs-helloworld]$ sudo firewall-cmd --zone=public --permanent --list-services
dhcpv6-client docker-registry ssh

docker info:

[[email protected] nodejs-helloworld]$ docker info
Containers: 6
 Running: 6
 Paused: 0
 Stopped: 0
Images: 25
Server Version: 1.12.5
Storage Driver: overlay
 Backing Filesystem: xfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-514.2.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.702 GiB
Name: localhost.localdomain
ID: SIYH:S5LU:BK4F:4BQW:FQOX:UP6D:HKS4:PXZH:DYTC:ASNF:FTBD:2ARV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 192.168.56.105:5000
 127.0.0.0/8

I’m running gitlab, drone, the registry, and all supporting services in containers:

[[email protected] nodejs-helloworld]$ docker ps
CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS                                                   NAMES
ed1e6befdb37        drone/drone:0.4              "/drone_static"          16 minutes ago      Up 16 minutes       0.0.0.0:8000->8000/tcp                                  drone
de84c9f2c6d5        sameersbn/postgresql:9.5-3   "/sbin/entrypoint.sh"    About an hour ago   Up About an hour    0.0.0.0:25432->5432/tcp                                 drone-postgresql
865a62b78e29        sameersbn/gitlab:8.13.3      "/sbin/entrypoint.sh "   About an hour ago   Up About an hour    443/tcp, 0.0.0.0:10022->22/tcp, 0.0.0.0:10080->80/tcp   gitlab
26e7bdbb9d2c        sameersbn/postgresql:9.5-3   "/sbin/entrypoint.sh"    About an hour ago   Up About an hour    0.0.0.0:15432->5432/tcp                                 gitlab-postgresql
155c77a8ebaa        redis:3.0.1                  "/entrypoint.sh redis"   About an hour ago   Up About an hour    0.0.0.0:16379->6379/tcp                                 gitlab-redis
f626208a118f        ephemeral/registry:2.5.1     "/entrypoint.sh serve"   2 days ago          Up 2 hours          0.0.0.0:5000->5000/tcp                                  registry

The repo is a simple nodejs helloworld app and its .drone.yml file is as follows:

[[email protected] nodejs-helloworld]$ cat .drone.yml
build:
  image: centos:7
  commands:
    - yum install -y epel-release
    - yum install -y nodejs npm
    - npm -d install
    - node_modules/mocha/bin/mocha

publish:
  docker:
    environment:
      - DOCKER_LAUNCH_DEBUG=true
    storage_driver: overlay
    insecure: true
    registry: 192.168.56.105:5000
    repo: 192.168.56.105:5000/ephemeral/nodejs-helloworld
    tags:
      - 1.0.0
      - "1.0"

If I build and push into the insecure registry from the command-line things work fine:

[[email protected] nodejs-helloworld]$ /usr/bin/docker build --pull=true --rm=true -f Dockerfile -t 192.168.56.105:5000/ephemeral/nodejs-helloworld:latest .
Sending build context to Docker daemon 67.07 kB
Step 1 : FROM node
latest: Pulling from library/node
75a822cd7888: Already exists
57de64c72267: Pull complete
4306be1e8943: Pull complete
871436ab7225: Pull complete
0110c26a367a: Pull complete
1f04fe713f1b: Pull complete
d8daeae79eeb: Pull complete
Digest: sha256:d63c3b05c7fc033b55698dce010922f6714ba44949521ddf5ab6bd69f22180b5
Status: Downloaded newer image for node:latest
 ---> d1699fb7d2bf
Step 2 : MAINTAINER nemonik
 ---> Running in a01b2c1a6ce1
 ---> fcf1270a5930
Removing intermediate container a01b2c1a6ce1
Step 3 : ENV NPM_CONFIG_LOGLEVEL warn
 ---> Running in 6d2c526514c8
 ---> c004236937e7
Removing intermediate container 6d2c526514c8
Step 4 : ADD . /app
 ---> 65e454e7aea6
Removing intermediate container 90e18b2ad9ba
Step 5 : EXPOSE 8080
 ---> Running in dc9523b55f1b
 ---> a61a90b3e46a
Removing intermediate container dc9523b55f1b
Step 6 : CMD nodejs /app/server.js
 ---> Running in f8232f768033
 ---> 4df14b191b3e
Removing intermediate container f8232f768033
Successfully built 4df14b191b3e
[[email protected] nodejs-helloworld]$ /usr/bin/docker push 192.168.56.105:5000/ephemeral/nodejs-helloworld:latest
The push refers to a repository [192.168.56.105:5000/ephemeral/nodejs-helloworld]
3a1ba67adcc7: Pushed
a05ad5eac50b: Pushed
604c78617f34: Pushed
fa18e5ffd316: Pushed
0a5e2b2ddeaa: Pushed
53c779688d06: Pushed
60a0858edcd5: Pushed
b6ca02dfe5e6: Pushed
latest: digest: sha256:65e5c03e13bce7fe8200f3710ba9dcb846bbcefd87102ceeaae051e589d39f80 size: 2004

I start the drone container like so:

docker run --name drone -d --publish 8000:8000 --restart on-failure:10 --env 'GIN_MODE=release' --env 'SERVER_ADDR=:8000' --env 'REMOTE_DRIVER=gitlab' --env 'REMOTE_CONFIG=http://192.168.56.105:10080?client_id=8798980845055d4d57c163872ab571b8f3fcc49c244dc115f531243acca03157&client_secret=2f1ae6114df42744406fdec5f8beeb7959454893c7e8c6f2d26c1087f4ffe63e&open=true' --env 'DATABASE_DRIVER=postgres' --env 'DATABASE_CONFIG=postgres://drone:[email protected]:25432/drone?sslmode=disable' --volume /var/run/docker.sock:/var/run/docker.sock --volume /srv/docker/drone/var/lib/drone drone/drone:0.4

From drone-docker plugin, the container image builds fine, but won’t push erroring out with dial tcp 192.168.56.105:5000: getsockopt: no route to host.

The logs for the drone-docker plugin are as follows:

[info] Pulling image plugins/drone-docker:latest
time="2016-12-29T18:13:40Z" level=debug msg="Launch config dockerlaunch.Config{Fork:false, PidOne:false, CommandName:\"\", DnsConfig:netconf.DnsConfig{Override:false, Nameservers:[]string(nil), Search:[]string(nil)}, BridgeName:\"\", BridgeAddress:\"\", BridgeMtu:0, CgroupHierarchy:map[string]string(nil), LogFile:\"\", NoLog:false, EmulateSystemd:false, NoFiles:0x0, Environment:[]string(nil), GraphDirectory:\"\"}" 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting devtmpfs /dev devtmpfs " 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting none /dev/pts devpts " 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting shm /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k" 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime" 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting none /proc proc " 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting none /run tmpfs " 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting none /sys sysfs " 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting none /sys/fs/cgroup tmpfs " 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting none /sys/fs/selinux selinuxfs " 
time="2016-12-29T18:13:40Z" level=debug msg="/proc/cgroups: #subsys_name\thierarchy\tnum_cgroups\tenabled" 
time="2016-12-29T18:13:40Z" level=debug msg="/proc/cgroups: cpuset\t3\t11\t1" 
time="2016-12-29T18:13:40Z" level=debug msg="/proc/cgroups: cpu\t8\t76\t1" 
time="2016-12-29T18:13:40Z" level=debug msg="/proc/cgroups: cpuacct\t8\t76\t1" 
time="2016-12-29T18:13:40Z" level=debug msg="/proc/cgroups: memory\t5\t76\t1" 
time="2016-12-29T18:13:40Z" level=debug msg="/proc/cgroups: devices\t11\t76\t1" 
time="2016-12-29T18:13:40Z" level=debug msg="/proc/cgroups: freezer\t4\t11\t1" 
time="2016-12-29T18:13:40Z" level=debug msg="/proc/cgroups: net_cls\t6\t11\t1" 
time="2016-12-29T18:13:40Z" level=debug msg="/proc/cgroups: blkio\t9\t76\t1" 
time="2016-12-29T18:13:40Z" level=debug msg="/proc/cgroups: perf_event\t7\t11\t1" 
time="2016-12-29T18:13:40Z" level=debug msg="/proc/cgroups: hugetlb\t2\t11\t1" 
time="2016-12-29T18:13:40Z" level=debug msg="/proc/cgroups: pids\t10\t11\t1" 
time="2016-12-29T18:13:40Z" level=debug msg="/proc/cgroups: net_prio\t6\t11\t1" 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting none /sys/fs/cgroup/memory cgroup memory" 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting none /sys/fs/cgroup/devices cgroup devices" 
time="2016-12-29T18:13:40Z" level=debug msg="Creating /sys/fs/cgroup/net_cls,net_prio" 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting none /sys/fs/cgroup/net_cls,net_prio cgroup net_cls,net_prio" 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting none /sys/fs/cgroup/pids cgroup pids" 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting none /sys/fs/cgroup/perf_event cgroup perf_event" 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting none /sys/fs/cgroup/hugetlb cgroup hugetlb" 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting none /sys/fs/cgroup/cpuset cgroup cpuset" 
time="2016-12-29T18:13:40Z" level=debug msg="Creating /sys/fs/cgroup/cpu,cpuacct" 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting none /sys/fs/cgroup/cpu,cpuacct cgroup cpu,cpuacct" 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting none /sys/fs/cgroup/freezer cgroup freezer" 
time="2016-12-29T18:13:40Z" level=debug msg="Mounting none /sys/fs/cgroup/blkio cgroup blkio" 
time="2016-12-29T18:13:40Z" level=debug msg="Done mouting cgroupfs" 
time="2016-12-29T18:13:40Z" level=debug msg="Creating /tmp" 
time="2016-12-29T18:13:40Z" level=debug msg="Creating /root/.ssh" 
time="2016-12-29T18:13:40Z" level=debug msg="Symlinking /lib => usr/lib" 
time="2016-12-29T18:13:40Z" level=debug msg="Symlinking /sbin => usr/sbin" 
time="2016-12-29T18:13:40Z" level=debug msg="Symlinking /var/run => ../run" 
time="2016-12-29T18:13:40Z" level=debug msg="Creating /etc/ssl/certs" 
time="2016-12-29T18:13:40Z" level=debug msg="Launching Docker /go/bin/drone-docker docker [-- {\"workspace\":{\"root\":\"/drone/src\",\"path\":\"/drone/src/192.168.56.105/nemonik/nodejs-helloworld\",\"netrc\":{\"machine\":\"192.168.56.105:10080\",\"login\":\"drone-ci-token\",\"user\":\"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ0ZXh0IjoibmVtb25pay9ub2RlanMtaGVsbG93b3JsZCIsInR5cGUiOiJob29rIn0.jDBkfwSPh9zXB7U4ZV3ek939lAwjRKAvlGOG6Vfur7g\"},\"keys\":{\"public\":\"snip\",\"private\":\"snip\"}},\"system\":{\"version\":\"\",\"link_url\":\"http://192.168.56.105:8000\",\"plugins\":null,\"globals\":null,\"privileged_plugins\":null},\"repo\":{\"scm\":\"\",\"owner\":\"nemonik\",\"name\":\"nodejs-helloworld\",\"full_name\":\"nemonik/nodejs-helloworld\",\"avatar_url\":\"\",\"link_url\":\"http://192.168.56.105:10080/nemonik/nodejs-helloworld\",\"clone_url\":\"http://192.168.56.105:10080/nemonik/nodejs-helloworld.git\",\"default_branch\":\"master\",\"timeout\":60,\"private\":false,\"trusted\":false,\"allow_pr\":true,\"allow_push\":true,\"allow_deploys\":false,\"allow_tags\":false},\"build\":{\"number\":9,\"event\":\"push\",\"status\":\"running\",\"enqueued_at\":1483035159,\"created_at\":1483035159,\"started_at\":1483035159,\"finished_at\":0,\"deploy_to\":\"\",\"commit\":\"922dadd24f0aa36d94914997c3c03678a31b1762\",\"branch\":\"master\",\"ref\":\"refs/heads/master\",\"refspec\":\"\",\"remote\":\"\",\"title\":\"\",\"message\":\"Update .drone.yml\",\"timestamp\":0,\"author\":\"nemonik\",\"author_avatar\":\"https://www.gravatar.com/avatar/5b6ada663d675c03221a819f10d00587.jpg?s=128\",\"author_email\":\"[email protected]\",\"link_url\":\"\"},\"job\":{\"id\":9,\"number\":1,\"status\":\"running\",\"exit_code\":0,\"enqueued_at\":1483035159,\"started_at\":1483035159,\"finished_at\":0,\"environment\":{}},\"vargs\":{\"insecure\":true,\"registry\":\"192.168.56.105:5000\",\"repo\":\"192.168.56.105:5000/ephemeral/nodejs-helloworld\",\"storage_driver\":\"overlay\",\"tags\":[\"1.0.0\",\"1.0\"]}}]" 
Drone Docker Plugin built at 2016-07-29 06:41:09 UTC
$ /usr/bin/docker daemon -g /drone/docker -s overlay --insecure-registry 192.168.56.105:5000
time="2016-12-29T18:13:40.670913412Z" level=info msg="Graph migration to content-addressability took 0.00 seconds" 
time="2016-12-29T18:13:40.672775412Z" level=warning msg="Running modprobe bridge br_netfilter failed with message: modprobe: can't change directory to '/lib/modules': No such file or directory\n, error: exit status 1" 
time="2016-12-29T18:13:40.673871202Z" level=warning msg="Running modprobe nf_nat failed with message: `modprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1" 
time="2016-12-29T18:13:40.703281234Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.18.0.0/16. Daemon option --bip can be used to set a preferred IP address" 
time="2016-12-29T18:13:40.745039871Z" level=info msg="Daemon has completed initialization" 
time="2016-12-29T18:13:40.745066978Z" level=info msg="Docker daemon" commit=c3959b1 execdriver=native-0.2 graphdriver=overlay version=1.10.2 
time="2016-12-29T18:13:40.749711970Z" level=info msg="API listen on /var/run/docker.sock" 
A username was not specified. Assuming anonymous publishing.
$ /usr/bin/docker version
Client:
Version:      1.10.2
API version:  1.22
Go version:   go1.5.3
Git commit:   c3959b1
Built:        Mon Feb 22 22:37:33 2016
OS/Arch:      linux/amd64

Server:
Version:      1.10.2
API version:  1.22
Go version:   go1.5.3
Git commit:   c3959b1
Built:        Mon Feb 22 22:37:33 2016
OS/Arch:      linux/amd64
$ /usr/bin/docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.10.2
Storage Driver: overlay
Backing Filesystem: xfs
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
Volume: local
Network: null host bridge
Kernel Version: 3.10.0-514.2.2.el7.x86_64
Operating System: <unknown> (containerized)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.702 GiB
Name: ed11e1bf0f74
ID: 55GL:QYKM:EHN4:OKSZ:JB57:SZGT:IQIB:DCG4:S6SZ:33PD:ECEZ:3MML
Debug mode (server): true
File Descriptors: 10
Goroutines: 20
System Time: 2016-12-29T18:13:45.746590996Z
EventsListeners: 0
Init SHA1: 
Init Path: /usr/bin/docker
Docker Root Dir: /drone/docker
$ /usr/bin/docker build --pull=true --rm=true -f Dockerfile -t 192.168.56.105:5000/ephemeral/nodejs-helloworld:latest .
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Sending build context to Docker daemon 557.1 kB\rSending build context to Docker daemon 1.114 MB\rSending build context to Docker daemon 1.572 MB\r\r
Step 1 : FROM node
latest: Pulling from library/node
75a822cd7888: Pulling fs layer
57de64c72267: Pulling fs layer
4306be1e8943: Pulling fs layer
871436ab7225: Pulling fs layer
0110c26a367a: Pulling fs layer
1f04fe713f1b: Pulling fs layer
d8daeae79eeb: Pulling fs layer
871436ab7225: Waiting
0110c26a367a: Waiting
1f04fe713f1b: Waiting
d8daeae79eeb: Waiting
57de64c72267: Verifying Checksum
57de64c72267: Download complete
4306be1e8943: Verifying Checksum
4306be1e8943: Download complete
0110c26a367a: Verifying Checksum
0110c26a367a: Download complete
75a822cd7888: Verifying Checksum
75a822cd7888: Download complete
1f04fe713f1b: Verifying Checksum
1f04fe713f1b: Download complete
75a822cd7888: Pull complete
75a822cd7888: Pull complete
57de64c72267: Pull complete
57de64c72267: Pull complete
4306be1e8943: Pull complete
4306be1e8943: Pull complete
d8daeae79eeb: Verifying Checksum
d8daeae79eeb: Download complete
871436ab7225: Verifying Checksum
871436ab7225: Download complete
871436ab7225: Pull complete
871436ab7225: Pull complete
0110c26a367a: Pull complete
0110c26a367a: Pull complete
1f04fe713f1b: Pull complete
1f04fe713f1b: Pull complete
d8daeae79eeb: Pull complete
d8daeae79eeb: Pull complete
Digest: sha256:d63c3b05c7fc033b55698dce010922f6714ba44949521ddf5ab6bd69f22180b5
Status: Downloaded newer image for node:latest
---> d1699fb7d2bf
Step 2 : MAINTAINER nemonik
---> Running in c6d4693271f7
time="2016-12-29T18:16:22.583200778Z" level=info msg="Layer sha256:504ad6dbe64007c0161b422fde744c2e3d6b18ba8a869030b443feeb94e0c318 cleaned up" 
---> 094d32198647
Removing intermediate container c6d4693271f7
Step 3 : ENV NPM_CONFIG_LOGLEVEL warn
---> Running in af819c547ae1
time="2016-12-29T18:16:25.469008305Z" level=info msg="Layer sha256:504ad6dbe64007c0161b422fde744c2e3d6b18ba8a869030b443feeb94e0c318 cleaned up" 
---> 17d0bd72fdd4
Removing intermediate container af819c547ae1
Step 4 : ADD . /app
---> 57d46888dcfc
Removing intermediate container 0314107847ad
Step 5 : EXPOSE 8080
---> Running in d85a4b56cf15
time="2016-12-29T18:16:30.752906781Z" level=info msg="Layer sha256:47c5229b36286b26cae7f37e317c399f16317dad4840e33e7ffa610397a7d6d2 cleaned up" 
---> 6a23f443573d
Removing intermediate container d85a4b56cf15
Step 6 : CMD nodejs /app/server.js
---> Running in 725282c8110f
time="2016-12-29T18:16:33.463625721Z" level=info msg="Layer sha256:47c5229b36286b26cae7f37e317c399f16317dad4840e33e7ffa610397a7d6d2 cleaned up" 
---> b24e6b7775c2
Removing intermediate container 725282c8110f
Successfully built b24e6b7775c2
$ /usr/bin/docker push 192.168.56.105:5000/ephemeral/nodejs-helloworld:latest
The push refers to a repository [192.168.56.105:5000/ephemeral/nodejs-helloworld]
Put http://192.168.56.105:5000/v1/repositories/ephemeral/nodejs-helloworld/: dial tcp 192.168.56.105:5000: getsockopt: no route to host
[info] build failed (exit code 1)

I’ve pushed this as far as I can go, and would appreciate help.

-Michael

I’m not sure that the registry being insecure is the root cause here. Based on the error message it sounds like a networking issue:

dial tcp 192.168.56.105:5000: getsockopt: no route to host

The docker plugin is a container an therefore has its own private network. Wouldn’t 192.168.56.105 point to the docker plugin’s own network?

@bradrydzewski, I got it to work… It appears the host’s firewall was blocking.

I had run

sudo firewall-cmd --zone=public --permanent --add-service=docker-registry

But until I stopped and restarted firewalld, it seems the firewall continued to block.

Hey, appreciate the response. Hopefully, this helps someone who comes behind me.

hi brother
I had the same problem but I didn’t turn on the firewall