[Solved] More than one private ip subnet on LAN interface?

lifeboy

I have a setup as per the attached diagram. I need to be able to access the devices on the 172.16.5.0/24 network via the 172.16.10.0/24 interface. On layer 2 these two are on the same switch, so I'm thinking that I can simply add a virtual IP address (172.16.10.254) to the LAN port and then I will have access to 172.16.10.1. After all the NIC's are both part of NODEA and I have ip routing enabled on the Proxmox OS (debian).

On the Proxmox node I have:

FT1-NodeA:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens6f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP group default qlen 1000
    link/ether ac:1f:6b:c5:95:20 brd ff:ff:ff:ff:ff:ff
3: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether ac:1f:6b:ca:e3:c8 brd ff:ff:ff:ff:ff:ff
4: ens7f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether ac:1f:6b:ca:e3:c9 brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.1/24 scope global ens7f1
       valid_lft forever preferred_lft forever
    inet6 fe80::ae1f:6bff:feca:e3c9/64 scope link 
       valid_lft forever preferred_lft forever
5: ens6f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether ac:1f:6b:c5:95:21 brd ff:ff:ff:ff:ff:ff
    inet 172.16.10.1/24 scope global ens6f1
       valid_lft forever preferred_lft forever
    inet 172.16.5.20/24 scope global ens6f1
       valid_lft forever preferred_lft forever
    inet6 fe80::ae1f:6bff:fec5:9521/64 scope link 
       valid_lft forever preferred_lft forever
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ac:1f:6b:ca:e3:c8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.131.1/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::ae1f:6bff:feca:e3c8/64 scope link 
       valid_lft forever preferred_lft forever
7: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ac:1f:6b:c5:95:20 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ae1f:6bff:fec5:9520/64 scope link 
       valid_lft forever preferred_lft forever
8: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 56:04:a3:29:50:04 brd ff:ff:ff:ff:ff:ff
9: tap100i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether 86:85:fe:de:86:81 brd ff:ff:ff:ff:ff:ff
10: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether 32:09:6b:2b:53:9c brd ff:ff:ff:ff:ff:ff
11: tap103i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether aa:cc:d8:f5:6c:27 brd ff:ff:ff:ff:ff:ff

FT1-NodeA:~# ip route show
default via 192.168.131.254 dev vmbr0 proto kernel onlink 
10.10.10.0/24 dev ens7f1 proto kernel scope link src 10.10.10.1 
172.16.5.0/24 dev ens6f1 proto kernel scope link src 172.16.5.20 
172.16.10.0/24 dev ens6f1 proto kernel scope link src 172.16.10.1 
192.168.131.0/24 dev vmbr0 proto kernel scope link src 192.168.131.1

FT1-NodeA:~# ping 172.16.10.254
PING 172.16.10.254 (172.16.10.254) 56(84) bytes of data.
^C
--- 172.16.10.254 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 9ms

However, on pfSense I have:

[2.4.4-RELEASE][roland@pfSense1A]/home/roland: ifconfig
vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=c00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
	ether ba:31:fd:37:4a:54
	hwaddr ba:31:fd:37:4a:54
	inet6 fe80::b831:fdff:fe37:4a54%vtnet0 prefixlen 64 scopeid 0x1 
	inet 192.168.131.254 netmask 0xffffff00 broadcast 192.168.131.255 
	inet 172.16.10.254 netmask 0xffffff00 broadcast 172.16.10.255 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet 10Gbase-T <full-duplex>
	status: active
vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=c00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
	ether a6:58:78:63:a2:62
	hwaddr a6:58:78:63:a2:62
	inet6 fe80::a458:78ff:fe63:a262%vtnet1 prefixlen 64 scopeid 0x2 
	inet 192.168.88.254 netmask 0xffffff00 broadcast 192.168.88.255 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet 10Gbase-T <full-duplex>
	status: active
enc0: flags=0<> metric 0 mtu 1536
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: enc 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: lo 
pflog0: flags=100<PROMISC> metric 0 mtu 33160
	groups: pflog 
pfsync0: flags=0<> metric 0 mtu 1500
	groups: pfsync 
	syncpeer: 224.0.0.240 maxupd: 128 defer: on
	syncok: 1

(The private IP range on the WAN port is only for setup purposes)

[2.4.4-RELEASE][roland@pfSense1A]/home/roland: ping 172.16.10.20
PING 172.16.10.20 (172.16.10.20): 56 data bytes
^C
--- 172.16.10.20 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
[2.4.4-RELEASE][roland@pfSense1A]/home/roland: ping 172.16.10.1
PING 172.16.10.1 (172.16.10.1): 56 data bytes
^C
--- 172.16.10.1 ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss

What am I missing? Why is this not working?

JKnott

@lifeboy

While it's possible to add an alias address, I can't understand, from your diagram, how you plan to do it. I see 2 switches with ens6f1 and ILOM (whatever that is) connected to 1. I don't see that 172.16.10.1 connected to any switch. How is it connected.

lifeboy

@JKnott said in More than one private ip subnet on LAN interface?:

@lifeboy

While it's possible to add an alias address, I can't understand, from your diagram, how you plan to do it.

I have already done it as can be seen from the "ifconfig" output I posted

I see 2 switches with ens6f1 and ILOM (whatever that is) connected to 1. I don't see that 172.16.10.1 connected to any switch. How is it connected.

172.16.10.1 is on ens6f1 as shown in my diagram.

JKnott

@lifeboy

Start simple and work your way up. Connect that pfSense interface to a switch and a device from each of the subnets to the switch. See if your idea works with just those devices connected. Then start adding to the network while ensuring things still work. When it fails, you'll have your suspect.

lifeboy

@JKnott: It all works individually. I can ping all addresses that are connected from the ports as shown. Did you really think I was so lazy that I haven't done that? I've been at this for some days now.

lifeboy

To clarify the problem in more detail:

pfSense has the bridge vmbr0 assigned to the LAN port. The primary IP address on there is 192.168.131.254. (That's vtnet0 in pfSense). I can ping that from the command line from the pfSense console, from the Proxmox host node (192.168.131.1) and from a VM (192.168.131.100). Of course, that's to be expected.
I have added a virtual ip (172.16.10.254) to vtnet0. I can ping that address from the console in pfSense (again, it's to be expected). However, I cannot ping that from the Proxmox host despite the fact that I have 172.16.10.1 on that host and have additionally assigned 172.10.10.20 as an alias on vmbr0 (vtnet0 on pfSense).

Is pfSense blocking that traffic? I have added a rule to allow that traffic explicitly, but the default rule does make provision for that anyway.

FT1-NodeA:~# ping 172.16.10.254
PING 172.16.10.254 (172.16.10.254) 56(84) bytes of data.
From 172.16.10.1 icmp_seq=1 Destination Host Unreachable
From 172.16.10.1 icmp_seq=2 Destination Host Unreachable
From 172.16.10.1 icmp_seq=3 Destination Host Unreachable
^C

JKnott

@lifeboy said in More than one private ip subnet on LAN interface?:

@JKnott: It all works individually. I can ping all addresses that are connected from the ports as shown. Did you really think I was so lazy that I haven't done that? I've been at this for some days now.

If it works with just those devices and not when you build up that configuration, then something in that configuration is causing the problem. Also, if it works at all, then firewall rules are not the issue. It has to be something in the way you connected things up. That is why I said to build things up until it breaks, so that you have some idea as to the cause of the problem.

lifeboy

@JKnott said in More than one private ip subnet on LAN interface?:

That is why I said to build things up until it breaks, so that you have some idea as to the cause of the problem.

That's not helpful. I can't do without pfSense, it's a VM on a Proxmox node.
I can't do without the WAN, otherwise I have no internet.
Proxmox needs corosync and ceph (172.16.10.1 and 10.10.10.1).
So how do you suggest I "build things up"? What can I take away to start "building up" from?
What you're suggesting doesn't make sense to me.

The problem seems to be that the bridge is not telling the host OS that there's an alias on that NIC bridge. The address 172.16.10.254 is functional on pfSense on the bridge, but on debian on the host it's not.

Is that a pfSense bridge problem, or is debian/proxmox?

From proxmox the bridge has:

6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ac:1f:6b:ca:e3:c8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.131.1/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::ae1f:6bff:feca:e3c8/64 scope link 
       valid_lft forever preferred_lft forever

However, from pfSense the bridge has:

vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=c00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
	ether ba:31:fd:37:4a:54
	hwaddr ba:31:fd:37:4a:54
	inet6 fe80::b831:fdff:fe37:4a54%vtnet0 prefixlen 64 scopeid 0x1 
	inet 192.168.131.254 netmask 0xffffff00 broadcast 192.168.131.255 
	inet 172.16.10.254 netmask 0xffffff00 broadcast 172.16.10.255 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet 10Gbase-T <full-duplex>
	status: active

Derelict

If all of those services are on the same broadcast domain why bother complicating things with different IP subnets? Just put everything on the same IP network.

Else, do the right thing and make an interface on pfSense for all of the different IP subnets.

lifeboy

@Derelict said in More than one private ip subnet on LAN interface?:

If all of those services are on the same broadcast domain why bother complicating things with different IP subnets? Just put everything on the same IP network.

I'm hoping that I can filter traffic to the other addresses (the 172.16.10.x and 5.x subnets) once I get the communication working.

Else, do the right thing and make an interface on pfSense for all of the different IP subnets.

I don't know how I can make more interfaces on pfSense if I don't have more NICs. That is why I added a virtual address to make it work. Would you mind giving a little more detail on how I could do that?

Derelict

It's virtual. Add another virtual NIC to the VM.

Putting multiple IP network ranges on the same interface is unsound design. It can be done temporarily to do something like renumber an interface but you can't really filter it as you would expect.

lifeboy

@Derelict Ah, excellent, thanks! Of course! I added an alias in the host node OS to create that NIC. I wasn't seeing the woods for the trees! :-)

lifeboy

This is not as simple I it seemed.

I added a macvlan type virtual nic to Proxmox (nodeB, just for testing).

:~# ip link add link vmbr0 vmbr2 address 00:11:11:11:11:12 type macvlan
:~# ip link set vmbr2 up

The new virtual address shows up:

:~# ip a show dev vmbr2
27: vmbr2@vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:11:11:11:11:12 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::211:11ff:fe11:1112/64 scope link 
       valid_lft forever preferred_lft forever

However, KVM doesn't recognise this. I added the device to the config, but cannot start pfSense:

:~# qm start 101
Can't exec "/usr/bin/ovs-vsctl": No such file or directory at /usr/share/perl5/PVE/Network.pm line 259.
can't add ovs port 'tap101i2'
kvm: network script /var/lib/qemu-server/pve-bridge failed with status 512
start failed: command '/usr/bin/kvm -id 101 -name pfSense1B -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=864821bf-3577-4964-aaba-3211b8689867' -smp '2,sockets=2,cores=1,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/101.vnc,password -cpu kvm64,+md-clear,+pcid,+aes,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 1024 -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'vmgenid,guid=cb39d484-3263-433e-afd7-77081ef99d30' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-vga,id=vga,bus=pci.0,addr=0x2' -chardev 'socket,path=/var/run/qemu-server/101.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:4a284d119617' -drive 'file=/mnt/pve/cephfs/template/iso/pfSense-CE-2.4.4-RELEASE-p3-amd64.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=rbd:speedy/vm-101-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/speedy.keyring,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=C6:71:28:74:A6:55,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -netdev 'type=tap,id=net1,ifname=tap101i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=5A:E0:03:17:40:F3,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=301' -netdev 'type=tap,id=net2,ifname=tap101i2,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=00:11:11:11:11:12,netdev=net2,bus=pci.0,addr=0x14,id=net2,bootindex=302' -machine 'type=pc+pve1'' failed: exit code 1

Proxmox doesn't show the interface in the GUI either. It's may be because it doesn't recognise it as a bridge? Do you know how I can fix this? I have read the macvlan man page, but still can't quite figure it out.

Derelict

No idea. I never use the CLI in proxmox. I have a vlan aware bridge consisting of a physical interface going to my switch. I just add an interface to a VM on that with the right VLAN tag and it just works.

johnpoz

@Derelict said in More than one private ip subnet on LAN interface?:

Putting multiple IP network ranges on the same interface is unsound design.

This is nice speak for utterly freaking borked ;)

JKnott

@johnpoz said in More than one private ip subnet on LAN interface?:

@Derelict said in More than one private ip subnet on LAN interface?:

Putting multiple IP network ranges on the same interface is unsound design.

This is nice speak for utterly freaking borked ;)

Yet entirely normal with IPv6. On the pfSense Router Advertisement page, it's quite easy to add multiple prefixes.

Derelict

That would be pertinent if we were talking about IPv6.

JKnott

@Derelict

Yet still doable with IPv4. I experimented with aliases on Linux several years ago. There's nothing to stop someone from having both a pubic and RFC 1918 address on an interface, for example.

lifeboy

Here's an updated diagram of what I'm trying to do.
I want to be able to reach "to here" "from here"

I have in the meantime figured out that I cannot "clone" a bridge to create a virtual bridge. I can however add a virtual NIC with ip link add link virt1 nic1 address xx:xx:xx:xx:xx:xx type macvlan, which then allows me to create a new bridge and then adding virt1 to the bridge. Once I add that to any KVM machine with an ip address in the 172.16.10.0/24 subnet, I'm then able to communicate with the other 172.16.10.0/24 machines. Of course, adding a 172.16.5.0/24 address allows me to reach the 172.16.5.0/24 devices.

Apart from above, is there a better way to achieve this? I only have to 2 x10G ports (ens6) and 2 x 25G ports (ens7) in these nodes and would like to work with these without added more NIC's.

Derelict

@lifeboy said in More than one private ip subnet on LAN interface? (emphasis added):

Apart from above, is there a better way to achieve this? I only have to 2 x10G ports (ens6) and 2 x 25G ports (ens7) in these nodes and would like to work with these without added more NIC's.

People use VLANs to achieve multiple broadcast domains presenting multiple virtual NICs to VM guests.

bond0 is an LACP LAGG to the switch for connectivity to the physical world.

Screen Shot 2019-12-24 at 11.35.41 AM.png

Screen Shot 2019-12-24 at 9.04.03 AM.png

Screen Shot 2019-12-24 at 8.53.06 AM.png

default            172.25.228.1       UGS      vtnet1
172.25.228.0/24    link#2             U        vtnet1 (WAN)
172.25.234.0/24    link#1             U        vtnet0 (LAN)
172.25.235.12/31   link#3             U        vtnet2 (OPT1)