[Solved] More than one private ip subnet on LAN interface?

lifeboy

@JKnott said in More than one private ip subnet on LAN interface?:

That is why I said to build things up until it breaks, so that you have some idea as to the cause of the problem.

That's not helpful. I can't do without pfSense, it's a VM on a Proxmox node.
I can't do without the WAN, otherwise I have no internet.
Proxmox needs corosync and ceph (172.16.10.1 and 10.10.10.1).
So how do you suggest I "build things up"? What can I take away to start "building up" from?
What you're suggesting doesn't make sense to me.

The problem seems to be that the bridge is not telling the host OS that there's an alias on that NIC bridge. The address 172.16.10.254 is functional on pfSense on the bridge, but on debian on the host it's not.

Is that a pfSense bridge problem, or is debian/proxmox?

From proxmox the bridge has:

6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ac:1f:6b:ca:e3:c8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.131.1/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::ae1f:6bff:feca:e3c8/64 scope link 
       valid_lft forever preferred_lft forever

However, from pfSense the bridge has:

vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=c00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
	ether ba:31:fd:37:4a:54
	hwaddr ba:31:fd:37:4a:54
	inet6 fe80::b831:fdff:fe37:4a54%vtnet0 prefixlen 64 scopeid 0x1 
	inet 192.168.131.254 netmask 0xffffff00 broadcast 192.168.131.255 
	inet 172.16.10.254 netmask 0xffffff00 broadcast 172.16.10.255 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet 10Gbase-T <full-duplex>
	status: active

Derelict

If all of those services are on the same broadcast domain why bother complicating things with different IP subnets? Just put everything on the same IP network.

Else, do the right thing and make an interface on pfSense for all of the different IP subnets.

lifeboy

@Derelict said in More than one private ip subnet on LAN interface?:

If all of those services are on the same broadcast domain why bother complicating things with different IP subnets? Just put everything on the same IP network.

I'm hoping that I can filter traffic to the other addresses (the 172.16.10.x and 5.x subnets) once I get the communication working.

Else, do the right thing and make an interface on pfSense for all of the different IP subnets.

I don't know how I can make more interfaces on pfSense if I don't have more NICs. That is why I added a virtual address to make it work. Would you mind giving a little more detail on how I could do that?

Derelict

It's virtual. Add another virtual NIC to the VM.

Putting multiple IP network ranges on the same interface is unsound design. It can be done temporarily to do something like renumber an interface but you can't really filter it as you would expect.

lifeboy

@Derelict Ah, excellent, thanks! Of course! I added an alias in the host node OS to create that NIC. I wasn't seeing the woods for the trees! :-)

lifeboy

This is not as simple I it seemed.

I added a macvlan type virtual nic to Proxmox (nodeB, just for testing).

:~# ip link add link vmbr0 vmbr2 address 00:11:11:11:11:12 type macvlan
:~# ip link set vmbr2 up

The new virtual address shows up:

:~# ip a show dev vmbr2
27: vmbr2@vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:11:11:11:11:12 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::211:11ff:fe11:1112/64 scope link 
       valid_lft forever preferred_lft forever

However, KVM doesn't recognise this. I added the device to the config, but cannot start pfSense:

:~# qm start 101
Can't exec "/usr/bin/ovs-vsctl": No such file or directory at /usr/share/perl5/PVE/Network.pm line 259.
can't add ovs port 'tap101i2'
kvm: network script /var/lib/qemu-server/pve-bridge failed with status 512
start failed: command '/usr/bin/kvm -id 101 -name pfSense1B -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=864821bf-3577-4964-aaba-3211b8689867' -smp '2,sockets=2,cores=1,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/101.vnc,password -cpu kvm64,+md-clear,+pcid,+aes,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 1024 -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'vmgenid,guid=cb39d484-3263-433e-afd7-77081ef99d30' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-vga,id=vga,bus=pci.0,addr=0x2' -chardev 'socket,path=/var/run/qemu-server/101.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:4a284d119617' -drive 'file=/mnt/pve/cephfs/template/iso/pfSense-CE-2.4.4-RELEASE-p3-amd64.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=rbd:speedy/vm-101-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/speedy.keyring,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=C6:71:28:74:A6:55,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -netdev 'type=tap,id=net1,ifname=tap101i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=5A:E0:03:17:40:F3,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=301' -netdev 'type=tap,id=net2,ifname=tap101i2,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=00:11:11:11:11:12,netdev=net2,bus=pci.0,addr=0x14,id=net2,bootindex=302' -machine 'type=pc+pve1'' failed: exit code 1

Proxmox doesn't show the interface in the GUI either. It's may be because it doesn't recognise it as a bridge? Do you know how I can fix this? I have read the macvlan man page, but still can't quite figure it out.

Derelict

No idea. I never use the CLI in proxmox. I have a vlan aware bridge consisting of a physical interface going to my switch. I just add an interface to a VM on that with the right VLAN tag and it just works.

johnpoz

@Derelict said in More than one private ip subnet on LAN interface?:

Putting multiple IP network ranges on the same interface is unsound design.

This is nice speak for utterly freaking borked ;)

JKnott

@johnpoz said in More than one private ip subnet on LAN interface?:

@Derelict said in More than one private ip subnet on LAN interface?:

Putting multiple IP network ranges on the same interface is unsound design.

This is nice speak for utterly freaking borked ;)

Yet entirely normal with IPv6. On the pfSense Router Advertisement page, it's quite easy to add multiple prefixes.

Derelict

That would be pertinent if we were talking about IPv6.

JKnott

@Derelict

Yet still doable with IPv4. I experimented with aliases on Linux several years ago. There's nothing to stop someone from having both a pubic and RFC 1918 address on an interface, for example.

lifeboy

Here's an updated diagram of what I'm trying to do.
I want to be able to reach "to here" "from here"

I have in the meantime figured out that I cannot "clone" a bridge to create a virtual bridge. I can however add a virtual NIC with ip link add link virt1 nic1 address xx:xx:xx:xx:xx:xx type macvlan, which then allows me to create a new bridge and then adding virt1 to the bridge. Once I add that to any KVM machine with an ip address in the 172.16.10.0/24 subnet, I'm then able to communicate with the other 172.16.10.0/24 machines. Of course, adding a 172.16.5.0/24 address allows me to reach the 172.16.5.0/24 devices.

Apart from above, is there a better way to achieve this? I only have to 2 x10G ports (ens6) and 2 x 25G ports (ens7) in these nodes and would like to work with these without added more NIC's.

Derelict

@lifeboy said in More than one private ip subnet on LAN interface? (emphasis added):

Apart from above, is there a better way to achieve this? I only have to 2 x10G ports (ens6) and 2 x 25G ports (ens7) in these nodes and would like to work with these without added more NIC's.

People use VLANs to achieve multiple broadcast domains presenting multiple virtual NICs to VM guests.

bond0 is an LACP LAGG to the switch for connectivity to the physical world.

Screen Shot 2019-12-24 at 11.35.41 AM.png

Screen Shot 2019-12-24 at 9.04.03 AM.png

Screen Shot 2019-12-24 at 8.53.06 AM.png

default            172.25.228.1       UGS      vtnet1
172.25.228.0/24    link#2             U        vtnet1 (WAN)
172.25.234.0/24    link#1             U        vtnet0 (LAN)
172.25.235.12/31   link#3             U        vtnet2 (OPT1)

johnpoz

@JKnott said in More than one private ip subnet on LAN interface?:

There's nothing to stop someone from having both a pubic and RFC 1918 address on an interface, for example.

Other than just plain common sense... There would be ZERO freaking reason to do such thing... Its not actually isolating anything and no point to it..

You can put as many IPs you want on the same L2 - doesn't mean it makes any sense, or you should do it, etc.

The reason for the link-local on IPv6 is to get information on its neighbors, its sends the RS from its link-local address, etc. etc.. There is all kinds of things that happen with the link-local in IPv6 that don't really need to get into here, and you if anyone should know anyway.

This is in no way the same as putting a public IPv4 and a rfc1918 IPv4 address on an interface - what does that accomplish other than complications and nonsense? If a device needs to have a rfc1918 and a public IPv4 then they should be on different L2s

Lets not mix in how things are done with IPv6 with the IPv4 talking about here.. Derelict has given the correct solution to the OP question, which is vlans!

lifeboy

The use of VLAN's is the better way to achieve this (as answered by @Derelict), but I wanted to add this to my own question as far as it pertains to creating virtual bridges:

It seems that one cannot add a virtual device to a NIC that is already part of a bridge.

:~# ip link add link ens7f0 virt1 address 00:11:22:33:44:55 type macvlan
RTNETLINK answers: Device or resource busy

However, if I create the Virtual NIC and then add it to a bridge, it works.

:~# ip link add link ens7f1 virt1 address 00:11:22:33:44:55 type macvlan
:~# brctl addbr virtb1
:~# brctl addif virtb1 virt1
:~# brctl show
bridge name   bridge id             STP enabled        interfaces
virtb1        8000.001122334455        no              virt1
vmbr0         8000.ac1f6bcae3e2        no              ens7f0
                                                       tap101i0
vmbr1         8000.ac1f6bc59544        no              ens6f0
                                                       tap101i1

Although Proxmox's GUI doesn't "see" the new bridge, if I edit the qemu conf file for the VM, the port is added and the VM starts.

Derelict

Sounds like you should probably move to a proxmox forum.