Linux Routing Two Interfaces on Same Subnet

December 16, 2017 admin

It's possible you will never need to do this and most likely there are experts that say avoid doing this. I recently had a challenge to do exactly this so I am recording my effort for future reference. This link helped me get it to work: https://access.redhat.com/solutions/30564

My setup is similar to the link above but a few more layers. My setup is a Centos7 VM under KVM. KVM using SR-IOV Network Virtual Functions. And to further complicate it the KVM hypervisor is an Oracle Cloud (OCI) bare metal server. OCI hands out additional public IP addresses using VNIC's which are added to the host via pass through. Out of scope here is adding VNIC's to KVM guests. Also note the public IP is natted to private IP's.

[root@centos7 opc]# cat /etc/iproute2/rt_tables 
[..]
100 t1
101 t2

[root@centos7 opc]# cat /etc/sysconfig/network-scripts/route-ens3
10.1.0.0/16 dev ens3 src 10.1.1.12 table t1
default via 10.1.1.1 dev ens3 table t1

[root@centos7 opc]# cat /etc/sysconfig/network-scripts/route-ens9
10.1.0.0/16 dev ens9 src 10.1.1.13 table t2
default via 10.1.1.1 dev ens9 table t2

[root@centos7 opc]# cat /etc/sysconfig/network-scripts/rule-ens3
table t1 from 10.1.1.12

[root@centos7 opc]# cat /etc/sysconfig/network-scripts/rule-ens9
table t2 from 10.1.1.13

Note may not need all below settings for example ens3 and ens9. Defaults may be enough.

[root@centos7 opc]# cat /etc/sysctl.d/99-sysctl.conf 
[..]
net.ipv4.conf.all.arp_filter = 1
net.ipv4.conf.default.arp_filter = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2

net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.ens3.rp_filter = 2
net.ipv4.conf.ens9.rp_filter = 2

Had some issues with /etc/sysconfig/network-scripts/route-ens* script not working at reboots, but manually running /etc/sysconfig/network-scripts/route-ens3 and route-ens9 worked. Commented DEFROUTE and GATEWAY and added NM_CONTROLLED=no and then routes worked at boot up.

[root@centos7 opc]# cat /etc/sysconfig/network-scripts/ifcfg-ens3
TYPE=Ethernet
BOOTPROTO=static
#DEFROUTE=yes
NAME=ens3
DEVICE=ens3
ONBOOT=yes
IPADDR=10.1.1.12
NETMASK=255.255.255.0
#GATEWAY=10.1.1.1
NM_CONTROLLED="no"

[root@centos7 opc]# cat /etc/sysconfig/network-scripts/ifcfg-ens9
TYPE=Ethernet
BOOTPROTO=static
#DEFROUTE=yes
NAME=ens9
DEVICE=ens9
ONBOOT=yes
IPADDR=10.1.1.13
NETMASK=255.255.255.0
#GATEWAY=10.1.1.1
NM_CONTROLLED="no"

Reboot

[opc@centos7 ~]$ sudo -s
[root@centos7 opc]# ip route show table t1
default via 10.1.1.1 dev ens3 
10.1.0.0/16 dev ens3 scope link src 10.1.1.12 

[root@centos7 opc]# ip route show table t2
default via 10.1.1.1 dev ens9 
10.1.0.0/16 dev ens9 scope link src 10.1.1.13 

[root@centos7 opc]# ip route show
10.1.1.0/24 dev ens3 proto kernel scope link src 10.1.1.12 
10.1.1.0/24 dev ens9 proto kernel scope link src 10.1.1.13 
169.254.0.0/16 dev ens3 scope link metric 1002 
169.254.0.0/16 dev ens9 scope link metric 1003 

[root@centos7 opc]# ping -I 10.1.1.12 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.1.1.12 : 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=61 time=10.6 ms

[root@centos7 opc]# ping -I 10.1.1.13 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.1.1.13 : 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=61 time=10.5 ms

Ping form hypervisor to VM IP's works now.

# ping 10.1.1.12
PING 10.1.1.12 (10.1.1.12) 56(84) bytes of data.
64 bytes from 10.1.1.12: icmp_seq=1 ttl=64 time=0.223 ms

# ping 10.1.1.13
PING 10.1.1.13 (10.1.1.13) 56(84) bytes of data.
64 bytes from 10.1.1.13: icmp_seq=1 ttl=64 time=0.189 ms