[Solved] More than one private ip subnet on LAN interface?

lifeboy

This is not as simple I it seemed.

I added a macvlan type virtual nic to Proxmox (nodeB, just for testing).

:~# ip link add link vmbr0 vmbr2 address 00:11:11:11:11:12 type macvlan
:~# ip link set vmbr2 up

The new virtual address shows up:

:~# ip a show dev vmbr2
27: vmbr2@vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:11:11:11:11:12 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::211:11ff:fe11:1112/64 scope link 
       valid_lft forever preferred_lft forever

However, KVM doesn't recognise this. I added the device to the config, but cannot start pfSense:

:~# qm start 101
Can't exec "/usr/bin/ovs-vsctl": No such file or directory at /usr/share/perl5/PVE/Network.pm line 259.
can't add ovs port 'tap101i2'
kvm: network script /var/lib/qemu-server/pve-bridge failed with status 512
start failed: command '/usr/bin/kvm -id 101 -name pfSense1B -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=864821bf-3577-4964-aaba-3211b8689867' -smp '2,sockets=2,cores=1,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/101.vnc,password -cpu kvm64,+md-clear,+pcid,+aes,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 1024 -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'vmgenid,guid=cb39d484-3263-433e-afd7-77081ef99d30' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-vga,id=vga,bus=pci.0,addr=0x2' -chardev 'socket,path=/var/run/qemu-server/101.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:4a284d119617' -drive 'file=/mnt/pve/cephfs/template/iso/pfSense-CE-2.4.4-RELEASE-p3-amd64.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=rbd:speedy/vm-101-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/speedy.keyring,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=C6:71:28:74:A6:55,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -netdev 'type=tap,id=net1,ifname=tap101i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=5A:E0:03:17:40:F3,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=301' -netdev 'type=tap,id=net2,ifname=tap101i2,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=00:11:11:11:11:12,netdev=net2,bus=pci.0,addr=0x14,id=net2,bootindex=302' -machine 'type=pc+pve1'' failed: exit code 1

Proxmox doesn't show the interface in the GUI either. It's may be because it doesn't recognise it as a bridge? Do you know how I can fix this? I have read the macvlan man page, but still can't quite figure it out.

Derelict

No idea. I never use the CLI in proxmox. I have a vlan aware bridge consisting of a physical interface going to my switch. I just add an interface to a VM on that with the right VLAN tag and it just works.

johnpoz

@Derelict said in More than one private ip subnet on LAN interface?:

Putting multiple IP network ranges on the same interface is unsound design.

This is nice speak for utterly freaking borked ;)

JKnott

@johnpoz said in More than one private ip subnet on LAN interface?:

@Derelict said in More than one private ip subnet on LAN interface?:

Putting multiple IP network ranges on the same interface is unsound design.

This is nice speak for utterly freaking borked ;)

Yet entirely normal with IPv6. On the pfSense Router Advertisement page, it's quite easy to add multiple prefixes.

Derelict

That would be pertinent if we were talking about IPv6.

JKnott

@Derelict

Yet still doable with IPv4. I experimented with aliases on Linux several years ago. There's nothing to stop someone from having both a pubic and RFC 1918 address on an interface, for example.

lifeboy

Here's an updated diagram of what I'm trying to do.
I want to be able to reach "to here" "from here"

I have in the meantime figured out that I cannot "clone" a bridge to create a virtual bridge. I can however add a virtual NIC with ip link add link virt1 nic1 address xx:xx:xx:xx:xx:xx type macvlan, which then allows me to create a new bridge and then adding virt1 to the bridge. Once I add that to any KVM machine with an ip address in the 172.16.10.0/24 subnet, I'm then able to communicate with the other 172.16.10.0/24 machines. Of course, adding a 172.16.5.0/24 address allows me to reach the 172.16.5.0/24 devices.

Apart from above, is there a better way to achieve this? I only have to 2 x10G ports (ens6) and 2 x 25G ports (ens7) in these nodes and would like to work with these without added more NIC's.

Derelict

@lifeboy said in More than one private ip subnet on LAN interface? (emphasis added):

Apart from above, is there a better way to achieve this? I only have to 2 x10G ports (ens6) and 2 x 25G ports (ens7) in these nodes and would like to work with these without added more NIC's.

People use VLANs to achieve multiple broadcast domains presenting multiple virtual NICs to VM guests.

bond0 is an LACP LAGG to the switch for connectivity to the physical world.

Screen Shot 2019-12-24 at 11.35.41 AM.png

Screen Shot 2019-12-24 at 9.04.03 AM.png

Screen Shot 2019-12-24 at 8.53.06 AM.png

default            172.25.228.1       UGS      vtnet1
172.25.228.0/24    link#2             U        vtnet1 (WAN)
172.25.234.0/24    link#1             U        vtnet0 (LAN)
172.25.235.12/31   link#3             U        vtnet2 (OPT1)

johnpoz

@JKnott said in More than one private ip subnet on LAN interface?:

There's nothing to stop someone from having both a pubic and RFC 1918 address on an interface, for example.

Other than just plain common sense... There would be ZERO freaking reason to do such thing... Its not actually isolating anything and no point to it..

You can put as many IPs you want on the same L2 - doesn't mean it makes any sense, or you should do it, etc.

The reason for the link-local on IPv6 is to get information on its neighbors, its sends the RS from its link-local address, etc. etc.. There is all kinds of things that happen with the link-local in IPv6 that don't really need to get into here, and you if anyone should know anyway.

This is in no way the same as putting a public IPv4 and a rfc1918 IPv4 address on an interface - what does that accomplish other than complications and nonsense? If a device needs to have a rfc1918 and a public IPv4 then they should be on different L2s

Lets not mix in how things are done with IPv6 with the IPv4 talking about here.. Derelict has given the correct solution to the OP question, which is vlans!

lifeboy

The use of VLAN's is the better way to achieve this (as answered by @Derelict), but I wanted to add this to my own question as far as it pertains to creating virtual bridges:

It seems that one cannot add a virtual device to a NIC that is already part of a bridge.

:~# ip link add link ens7f0 virt1 address 00:11:22:33:44:55 type macvlan
RTNETLINK answers: Device or resource busy

However, if I create the Virtual NIC and then add it to a bridge, it works.

:~# ip link add link ens7f1 virt1 address 00:11:22:33:44:55 type macvlan
:~# brctl addbr virtb1
:~# brctl addif virtb1 virt1
:~# brctl show
bridge name   bridge id             STP enabled        interfaces
virtb1        8000.001122334455        no              virt1
vmbr0         8000.ac1f6bcae3e2        no              ens7f0
                                                       tap101i0
vmbr1         8000.ac1f6bc59544        no              ens6f0
                                                       tap101i1

Although Proxmox's GUI doesn't "see" the new bridge, if I edit the qemu conf file for the VM, the port is added and the VM starts.

Derelict

Sounds like you should probably move to a proxmox forum.