本文是对docker 端口绑定到ipv6 导致ipv4请求无法转发的问题的一些记录和解决过程。
问题症状
vmware虚拟机的 centos 7 里面运行了docker 的mysql服务,端口转发3306:3306,结果发现虚拟机外的系统无法访问到虚拟机的3306的数据库服务。
核心的原因
docker 对与ipv6默认是没有打开forwarding 设置的
首先官方的介绍:
在默认的配置中,流量的端口转发分为两种:内部流量转发(本机),外部流量转发(跨机器)
举个例子:
#docker run -d -p 80:80 nginx
这个操作会在iptables中增加如下策略(是的,docker所有的端口转发都是靠iptables实现的)
#iptables -t nat -L -v
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER all -- any any anywhere anywhere ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
1 60 DOCKER all -- any any anywhere !loopback/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- any !docker0 172.18.0.0/16 anywhere
0 0 MASQUERADE all -- any !docker_gwbridge 172.19.0.0/16 anywhere
0 0 MASQUERADE tcp -- any any 172.18.0.2 172.18.0.2 tcp dpt:http
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- docker0 any anywhere anywhere
0 0 RETURN all -- docker_gwbridge any anywhere anywhere
0 0 DNAT tcp -- !docker0 any anywhere anywhere
防火墙的大致意思:所有的对本地的流量请求要走DOCKER这条链(chain)
所以只要请求到达Docker 这条链路,任何从docker0返回的数据会立即返回,因为在PREROUTING链中并没有相关docker0的流量控制
你还会发现有一个docker-proxy在运行,这用来处理来“桥”的请求的,docker-proxy会监听物理主机的80端口,然后将所有的请求转发到docker 容器中,这仅仅是本地的流量请求处理,不包含其他
你还会发现在DNAT的规则最后(Docker),任何来自80段口的流量,如果源地址不是来自docker0,会被转发到容器的ip地址的80段口,这一段,是为外部请求设定的,外部请求会直接转发到容器,不会经过任何的物理主机的network stack.
这样看来,通过命令curl localhost 足以来验证我们的容器是否可用来,或者也可以使用netstat来看一下docker-proxy是否监听到tcp4上,但是,这只能证明我们的容器,本地可用,仅仅本地
其实我们的docker-proxy的策略可以更加深层的设置为: 仅仅转发 源ip是目标ip并且段口是指定桥的名字的,但是这会额外增加特别多的防火墙规则,而且有一个kernel bug 和这个相关,所以我们默认是关闭的
#netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 :::2377 :::* LISTEN 7814/dockerd
tcp6 0 0 :::7946 :::* LISTEN 7814/dockerd
tcp6 0 0 :::80 :::* LISTEN 8503/docker-proxy
udp 0 0 0.0.0.0:4789 0.0.0.0:* -
udp6 0 0 :::7946 :::* 7814/dockerd
尝试使用curl命令检测端口请求内容:
#curl 127.0.0.1 # use the actual v4 address to make sure localhost
doesn't resolve to a v6 address
<!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html> root@e426b2319215:/go/src/github.com/docker/docker# curl 127.0.0.1 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html>
也可以通过lsof来判定:
# lsof -n -i TCP
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
dockerd 7814 root 13u IPv6 5955398 0t0 TCP *:2377 (LISTEN)
dockerd 7814 root 14u IPv4 5955400 0t0 TCP 172.17.0.2:55708->172.17.0.2:2377 (ESTABLISHED)
dockerd 7814 root 18u IPv6 5955401 0t0 TCP 172.17.0.2:2377->172.17.0.2:55708 (ESTABLISHED)
dockerd 7814 root 21u IPv6 5955403 0t0 TCP *:7946 (LISTEN)
docker-pr 8503 root 4u IPv6 5957159 0t0 TCP *:http (LISTEN)
其实我们有两种方案解决:
- 关闭ipv6 ubuntu:
$ nano /etc/default/grub ... GRUB_CMDLINE_LINUX="ipv6.disable=1" ... $ update-grub $ reboot
注意: 若为 ubuntu 14 : 需要vim 编辑 /etc/sysctl.conf 文件
net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1
然后执行:
sysctl -p
2:打开ipv6的forwarding
#临时修改方式,重启后本命令失效 sysctl -w net.ipv6.conf.all.forwarding=1
永久的修改方式,重启后仍有效: 编辑/etc/sysctl.conf 文件,给文件追加内容:
sysctl -w net.ipv6.conf.all.forwarding=1
然后执行
sysctl -p /etc/sysctl.conf
参考链接
[1] https://access.redhat.com/solutions/3114021
[2] https://github.com/moby/moby/issues/2174#issuecomment-289049493