cannot setup routing within connected containers

[nagylzs]

I have two docker networks:

docker network create --attachable --internal --subnet 10.241.32.0/24 nw_01_02
docker network create --attachable --internal --subnet 10.241.33.0/24 nw_02_03

I also have three docker containers:

• node01 10.241.32.12 on nw_01_02
• node02 10.241.32.11 on 10.241.33.11 and nw_01_02 és nw_02_03
• node01 10.241.33.12 on nw_02_03

I would like node02 to be a router between node01 and node03.

1. node01 is started with --network nw_01_02 --ip 10.241.32.12 --cap-add=NET_ADMIN
2. node03 is started with --network nw_02_03 --ip 10.241.33.12 --cap-add=NET_ADMIN
3. node02 (the gateway) is started on both networks with addresses 10.241.32.11 and 10.241.33.11 and also with
   --sysctl=net.ipv4.conf.all.src_valid_mark=1 --sysctl=net.ipv4.ip_forward=1 --cap-add=NET_ADMIN
4. on node01 we have
   ip route add 10.241.33.0/24 via 10.241.32.11
5. on node03 we have
   ip route add 10.241.32.0/24 via 10.241.33.11
   
   

I have create a minimal working example for this question, you can find it here: https://github.com/nagylzs/docker_routing_test

You can start the test this way:

git clone https://github.com/nagylzs/docker_routing_test.git
cd docker_routing_test

# create networks
./init.sh

# start three nodes, do this in three different terminals
./start01.sh
./start02.sh
./start03.sh

# add node02 to both networks, assign ip addresses
./connect_02.sh

#
# AT THIS POINT YOU CAN TRY ACCESSING node03 from node01 and vice versa
#

# cleanup: delete all containers and networks
./cleanup.sh

Here is what I see:

• node01 (10.241.32.12) can ping node02 on nw_01_02 (10.241.32.11)
• node03 (10.241.33.12) can ping node02 on nw_02_03 (10.241.33.11)
• node02 can ping any node on any address
• node01 (10.241.32.12) cannot ping node02 on nw_02_03 (10.241.33.11)
• node03 (10.241.33.12) cannot ping node02 on nw_01_02 (10.241.32.11)

So it seems that the gateway is not forwarding packets.

I have run tcpdump and wireshark on all of them, and here is what I have found:

• when node01 (10.241.32.12) sends IMCP PING to node02 on nw_01_02 (10.241.32.11), then it goes out on eth0 of node01, and it comes in on eth0 of node02. The reply comes back OK.
• when node01 (10.241.32.12) sends IMCP PING to node02 on nw_02_03 (10.241.33.11), then it goes out on eth0 of node01, and it DOES NOT COME IN on eth0 of node02.

This last thing I cannot understand. The nw_01_02 internal network has driver=bridge. I can see that the outgoing ICMP packet is put into an ethernet frame, and it is written out on eth0 on node02. But it never appears on eth0 on node02.

There are firewall rules on these nodes, but they default to INPUT ACCEPT and FORWARD accept.

Can anybody help me please? What am I doing wrong?

 

[nagylzs]

FYI I gave up on this.