[Solved] More than one private ip subnet on LAN interface?



  • I have a setup as per the attached diagram. I need to be able to access the devices on the 172.16.5.0/24 network via the 172.16.10.0/24 interface. On layer 2 these two are on the same switch, so I'm thinking that I can simply add a virtual IP address (172.16.10.254) to the LAN port and then I will have access to 172.16.10.1. After all the NIC's are both part of NODEA and I have ip routing enabled on the Proxmox OS (debian).

    On the Proxmox node I have:

    FT1-NodeA:~# ip a
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host 
           valid_lft forever preferred_lft forever
    2: ens6f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP group default qlen 1000
        link/ether ac:1f:6b:c5:95:20 brd ff:ff:ff:ff:ff:ff
    3: ens7f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
        link/ether ac:1f:6b:ca:e3:c8 brd ff:ff:ff:ff:ff:ff
    4: ens7f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
        link/ether ac:1f:6b:ca:e3:c9 brd ff:ff:ff:ff:ff:ff
        inet 10.10.10.1/24 scope global ens7f1
           valid_lft forever preferred_lft forever
        inet6 fe80::ae1f:6bff:feca:e3c9/64 scope link 
           valid_lft forever preferred_lft forever
    5: ens6f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
        link/ether ac:1f:6b:c5:95:21 brd ff:ff:ff:ff:ff:ff
        inet 172.16.10.1/24 scope global ens6f1
           valid_lft forever preferred_lft forever
        inet 172.16.5.20/24 scope global ens6f1
           valid_lft forever preferred_lft forever
        inet6 fe80::ae1f:6bff:fec5:9521/64 scope link 
           valid_lft forever preferred_lft forever
    6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
        link/ether ac:1f:6b:ca:e3:c8 brd ff:ff:ff:ff:ff:ff
        inet 192.168.131.1/24 scope global vmbr0
           valid_lft forever preferred_lft forever
        inet6 fe80::ae1f:6bff:feca:e3c8/64 scope link 
           valid_lft forever preferred_lft forever
    7: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
        link/ether ac:1f:6b:c5:95:20 brd ff:ff:ff:ff:ff:ff
        inet6 fe80::ae1f:6bff:fec5:9520/64 scope link 
           valid_lft forever preferred_lft forever
    8: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
        link/ether 56:04:a3:29:50:04 brd ff:ff:ff:ff:ff:ff
    9: tap100i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
        link/ether 86:85:fe:de:86:81 brd ff:ff:ff:ff:ff:ff
    10: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
        link/ether 32:09:6b:2b:53:9c brd ff:ff:ff:ff:ff:ff
    11: tap103i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
        link/ether aa:cc:d8:f5:6c:27 brd ff:ff:ff:ff:ff:ff
    
    FT1-NodeA:~# ip route show
    default via 192.168.131.254 dev vmbr0 proto kernel onlink 
    10.10.10.0/24 dev ens7f1 proto kernel scope link src 10.10.10.1 
    172.16.5.0/24 dev ens6f1 proto kernel scope link src 172.16.5.20 
    172.16.10.0/24 dev ens6f1 proto kernel scope link src 172.16.10.1 
    192.168.131.0/24 dev vmbr0 proto kernel scope link src 192.168.131.1 
    
    FT1-NodeA:~# ping 172.16.10.254
    PING 172.16.10.254 (172.16.10.254) 56(84) bytes of data.
    ^C
    --- 172.16.10.254 ping statistics ---
    2 packets transmitted, 0 received, 100% packet loss, time 9ms
    

    However, on pfSense I have:

    [2.4.4-RELEASE][roland@pfSense1A]/home/roland: ifconfig
    vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    	options=c00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
    	ether ba:31:fd:37:4a:54
    	hwaddr ba:31:fd:37:4a:54
    	inet6 fe80::b831:fdff:fe37:4a54%vtnet0 prefixlen 64 scopeid 0x1 
    	inet 192.168.131.254 netmask 0xffffff00 broadcast 192.168.131.255 
    	inet 172.16.10.254 netmask 0xffffff00 broadcast 172.16.10.255 
    	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    	media: Ethernet 10Gbase-T <full-duplex>
    	status: active
    vtnet1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    	options=c00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
    	ether a6:58:78:63:a2:62
    	hwaddr a6:58:78:63:a2:62
    	inet6 fe80::a458:78ff:fe63:a262%vtnet1 prefixlen 64 scopeid 0x2 
    	inet 192.168.88.254 netmask 0xffffff00 broadcast 192.168.88.255 
    	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    	media: Ethernet 10Gbase-T <full-duplex>
    	status: active
    enc0: flags=0<> metric 0 mtu 1536
    	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    	groups: enc 
    lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    	inet6 ::1 prefixlen 128 
    	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 
    	inet 127.0.0.1 netmask 0xff000000 
    	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    	groups: lo 
    pflog0: flags=100<PROMISC> metric 0 mtu 33160
    	groups: pflog 
    pfsync0: flags=0<> metric 0 mtu 1500
    	groups: pfsync 
    	syncpeer: 224.0.0.240 maxupd: 128 defer: on
    	syncok: 1
    

    (The private IP range on the WAN port is only for setup purposes)

    [2.4.4-RELEASE][roland@pfSense1A]/home/roland: ping 172.16.10.20
    PING 172.16.10.20 (172.16.10.20): 56 data bytes
    ^C
    --- 172.16.10.20 ping statistics ---
    3 packets transmitted, 0 packets received, 100.0% packet loss
    [2.4.4-RELEASE][roland@pfSense1A]/home/roland: ping 172.16.10.1
    PING 172.16.10.1 (172.16.10.1): 56 data bytes
    ^C
    --- 172.16.10.1 ping statistics ---
    5 packets transmitted, 0 packets received, 100.0% packet loss
    

    What am I missing? Why is this not working?

    Diagram1.png



  • @lifeboy

    While it's possible to add an alias address, I can't understand, from your diagram, how you plan to do it. I see 2 switches with ens6f1 and ILOM (whatever that is) connected to 1. I don't see that 172.16.10.1 connected to any switch. How is it connected.



  • @JKnott said in More than one private ip subnet on LAN interface?:

    @lifeboy

    While it's possible to add an alias address, I can't understand, from your diagram, how you plan to do it.

    I have already done it as can be seen from the "ifconfig" output I posted

    I see 2 switches with ens6f1 and ILOM (whatever that is) connected to 1. I don't see that 172.16.10.1 connected to any switch. How is it connected.

    172.16.10.1 is on ens6f1 as shown in my diagram.



  • @lifeboy

    Start simple and work your way up. Connect that pfSense interface to a switch and a device from each of the subnets to the switch. See if your idea works with just those devices connected. Then start adding to the network while ensuring things still work. When it fails, you'll have your suspect.



  • @JKnott: It all works individually. I can ping all addresses that are connected from the ports as shown. Did you really think I was so lazy that I haven't done that? I've been at this for some days now.



  • To clarify the problem in more detail:

    1. pfSense has the bridge vmbr0 assigned to the LAN port. The primary IP address on there is 192.168.131.254. (That's vtnet0 in pfSense). I can ping that from the command line from the pfSense console, from the Proxmox host node (192.168.131.1) and from a VM (192.168.131.100). Of course, that's to be expected.

    2. I have added a virtual ip (172.16.10.254) to vtnet0. I can ping that address from the console in pfSense (again, it's to be expected). However, I cannot ping that from the Proxmox host despite the fact that I have 172.16.10.1 on that host and have additionally assigned 172.10.10.20 as an alias on vmbr0 (vtnet0 on pfSense).

    Is pfSense blocking that traffic? I have added a rule to allow that traffic explicitly, but the default rule does make provision for that anyway.

    FT1-NodeA:~# ping 172.16.10.254
    PING 172.16.10.254 (172.16.10.254) 56(84) bytes of data.
    From 172.16.10.1 icmp_seq=1 Destination Host Unreachable
    From 172.16.10.1 icmp_seq=2 Destination Host Unreachable
    From 172.16.10.1 icmp_seq=3 Destination Host Unreachable
    ^C
    


  • @lifeboy said in More than one private ip subnet on LAN interface?:

    @JKnott: It all works individually. I can ping all addresses that are connected from the ports as shown. Did you really think I was so lazy that I haven't done that? I've been at this for some days now.

    If it works with just those devices and not when you build up that configuration, then something in that configuration is causing the problem. Also, if it works at all, then firewall rules are not the issue. It has to be something in the way you connected things up. That is why I said to build things up until it breaks, so that you have some idea as to the cause of the problem.



  • @JKnott said in More than one private ip subnet on LAN interface?:

    That is why I said to build things up until it breaks, so that you have some idea as to the cause of the problem.

    That's not helpful. I can't do without pfSense, it's a VM on a Proxmox node.
    I can't do without the WAN, otherwise I have no internet.
    Proxmox needs corosync and ceph (172.16.10.1 and 10.10.10.1).
    So how do you suggest I "build things up"? What can I take away to start "building up" from?
    What you're suggesting doesn't make sense to me.

    The problem seems to be that the bridge is not telling the host OS that there's an alias on that NIC bridge. The address 172.16.10.254 is functional on pfSense on the bridge, but on debian on the host it's not.

    Is that a pfSense bridge problem, or is debian/proxmox?

    From proxmox the bridge has:

    6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
        link/ether ac:1f:6b:ca:e3:c8 brd ff:ff:ff:ff:ff:ff
        inet 192.168.131.1/24 scope global vmbr0
           valid_lft forever preferred_lft forever
        inet6 fe80::ae1f:6bff:feca:e3c8/64 scope link 
           valid_lft forever preferred_lft forever
    

    However, from pfSense the bridge has:

    vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    	options=c00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
    	ether ba:31:fd:37:4a:54
    	hwaddr ba:31:fd:37:4a:54
    	inet6 fe80::b831:fdff:fe37:4a54%vtnet0 prefixlen 64 scopeid 0x1 
    	inet 192.168.131.254 netmask 0xffffff00 broadcast 192.168.131.255 
    	inet 172.16.10.254 netmask 0xffffff00 broadcast 172.16.10.255 
    	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    	media: Ethernet 10Gbase-T <full-duplex>
    	status: active
    

  • LAYER 8 Netgate

    If all of those services are on the same broadcast domain why bother complicating things with different IP subnets? Just put everything on the same IP network.

    Else, do the right thing and make an interface on pfSense for all of the different IP subnets.



  • @Derelict said in More than one private ip subnet on LAN interface?:

    If all of those services are on the same broadcast domain why bother complicating things with different IP subnets? Just put everything on the same IP network.

    I'm hoping that I can filter traffic to the other addresses (the 172.16.10.x and 5.x subnets) once I get the communication working.

    Else, do the right thing and make an interface on pfSense for all of the different IP subnets.

    I don't know how I can make more interfaces on pfSense if I don't have more NICs. That is why I added a virtual address to make it work. Would you mind giving a little more detail on how I could do that?


  • LAYER 8 Netgate

    It's virtual. Add another virtual NIC to the VM.

    Putting multiple IP network ranges on the same interface is unsound design. It can be done temporarily to do something like renumber an interface but you can't really filter it as you would expect.



  • @Derelict Ah, excellent, thanks! Of course! I added an alias in the host node OS to create that NIC. I wasn't seeing the woods for the trees! :-)



  • This is not as simple I it seemed.

    I added a macvlan type virtual nic to Proxmox (nodeB, just for testing).

    :~# ip link add link vmbr0 vmbr2 address 00:11:11:11:11:12 type macvlan
    :~# ip link set vmbr2 up
    

    The new virtual address shows up:

    :~# ip a show dev vmbr2
    27: vmbr2@vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
        link/ether 00:11:11:11:11:12 brd ff:ff:ff:ff:ff:ff
        inet6 fe80::211:11ff:fe11:1112/64 scope link 
           valid_lft forever preferred_lft forever
    

    However, KVM doesn't recognise this. I added the device to the config, but cannot start pfSense:

    :~# qm start 101
    Can't exec "/usr/bin/ovs-vsctl": No such file or directory at /usr/share/perl5/PVE/Network.pm line 259.
    can't add ovs port 'tap101i2'
    kvm: network script /var/lib/qemu-server/pve-bridge failed with status 512
    start failed: command '/usr/bin/kvm -id 101 -name pfSense1B -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=864821bf-3577-4964-aaba-3211b8689867' -smp '2,sockets=2,cores=1,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc unix:/var/run/qemu-server/101.vnc,password -cpu kvm64,+md-clear,+pcid,+aes,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 1024 -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'vmgenid,guid=cb39d484-3263-433e-afd7-77081ef99d30' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'virtio-vga,id=vga,bus=pci.0,addr=0x2' -chardev 'socket,path=/var/run/qemu-server/101.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:4a284d119617' -drive 'file=/mnt/pve/cephfs/template/iso/pfSense-CE-2.4.4-RELEASE-p3-amd64.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=rbd:speedy/vm-101-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/speedy.keyring,if=none,id=drive-virtio0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=C6:71:28:74:A6:55,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -netdev 'type=tap,id=net1,ifname=tap101i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=5A:E0:03:17:40:F3,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=301' -netdev 'type=tap,id=net2,ifname=tap101i2,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=00:11:11:11:11:12,netdev=net2,bus=pci.0,addr=0x14,id=net2,bootindex=302' -machine 'type=pc+pve1'' failed: exit code 1
    

    Proxmox doesn't show the interface in the GUI either. It's may be because it doesn't recognise it as a bridge? Do you know how I can fix this? I have read the macvlan man page, but still can't quite figure it out.


  • LAYER 8 Netgate

    No idea. I never use the CLI in proxmox. I have a vlan aware bridge consisting of a physical interface going to my switch. I just add an interface to a VM on that with the right VLAN tag and it just works.


  • LAYER 8 Global Moderator

    @Derelict said in More than one private ip subnet on LAN interface?:

    Putting multiple IP network ranges on the same interface is unsound design.

    This is nice speak for utterly freaking borked ;)



  • @johnpoz said in More than one private ip subnet on LAN interface?:

    @Derelict said in More than one private ip subnet on LAN interface?:

    Putting multiple IP network ranges on the same interface is unsound design.

    This is nice speak for utterly freaking borked ;)

    Yet entirely normal with IPv6. On the pfSense Router Advertisement page, it's quite easy to add multiple prefixes.


  • LAYER 8 Netgate

    That would be pertinent if we were talking about IPv6.



  • @Derelict

    Yet still doable with IPv4. I experimented with aliases on Linux several years ago. There's nothing to stop someone from having both a pubic and RFC 1918 address on an interface, for example.



  • Here's an updated diagram of what I'm trying to do. Diagram1.png
    I want to be able to reach "to here" "from here"

    I have in the meantime figured out that I cannot "clone" a bridge to create a virtual bridge. I can however add a virtual NIC with ip link add link virt1 nic1 address xx:xx:xx:xx:xx:xx type macvlan, which then allows me to create a new bridge and then adding virt1 to the bridge. Once I add that to any KVM machine with an ip address in the 172.16.10.0/24 subnet, I'm then able to communicate with the other 172.16.10.0/24 machines. Of course, adding a 172.16.5.0/24 address allows me to reach the 172.16.5.0/24 devices.

    Apart from above, is there a better way to achieve this? I only have to 2 x10G ports (ens6) and 2 x 25G ports (ens7) in these nodes and would like to work with these without added more NIC's.


  • LAYER 8 Netgate

    @lifeboy said in More than one private ip subnet on LAN interface? (emphasis added):

    Apart from above, is there a better way to achieve this? I only have to 2 x10G ports (ens6) and 2 x 25G ports (ens7) in these nodes and would like to work with these without added more NIC's.

    People use VLANs to achieve multiple broadcast domains presenting multiple virtual NICs to VM guests.

    bond0 is an LACP LAGG to the switch for connectivity to the physical world.

    Screen Shot 2019-12-24 at 11.35.41 AM.png

    Screen Shot 2019-12-24 at 9.04.03 AM.png

    Screen Shot 2019-12-24 at 8.53.06 AM.png

    default            172.25.228.1       UGS      vtnet1
    172.25.228.0/24    link#2             U        vtnet1 (WAN)
    172.25.234.0/24    link#1             U        vtnet0 (LAN)
    172.25.235.12/31   link#3             U        vtnet2 (OPT1)
    

  • LAYER 8 Global Moderator

    @JKnott said in More than one private ip subnet on LAN interface?:

    There's nothing to stop someone from having both a pubic and RFC 1918 address on an interface, for example.

    Other than just plain common sense... There would be ZERO freaking reason to do such thing... Its not actually isolating anything and no point to it..

    You can put as many IPs you want on the same L2 - doesn't mean it makes any sense, or you should do it, etc.

    The reason for the link-local on IPv6 is to get information on its neighbors, its sends the RS from its link-local address, etc. etc.. There is all kinds of things that happen with the link-local in IPv6 that don't really need to get into here, and you if anyone should know anyway.

    This is in no way the same as putting a public IPv4 and a rfc1918 IPv4 address on an interface - what does that accomplish other than complications and nonsense? If a device needs to have a rfc1918 and a public IPv4 then they should be on different L2s

    Lets not mix in how things are done with IPv6 with the IPv4 talking about here.. Derelict has given the correct solution to the OP question, which is vlans!



  • The use of VLAN's is the better way to achieve this (as answered by @Derelict), but I wanted to add this to my own question as far as it pertains to creating virtual bridges:

    It seems that one cannot add a virtual device to a NIC that is already part of a bridge.

    :~# ip link add link ens7f0 virt1 address 00:11:22:33:44:55 type macvlan
    RTNETLINK answers: Device or resource busy

    However, if I create the Virtual NIC and then add it to a bridge, it works.

    :~# ip link add link ens7f1 virt1 address 00:11:22:33:44:55 type macvlan
    :~# brctl addbr virtb1
    :~# brctl addif virtb1 virt1
    :~# brctl show
    bridge name   bridge id             STP enabled        interfaces
    virtb1        8000.001122334455        no              virt1
    vmbr0         8000.ac1f6bcae3e2        no              ens7f0
                                                           tap101i0
    vmbr1         8000.ac1f6bc59544        no              ens6f0
                                                           tap101i1
    

    Although Proxmox's GUI doesn't "see" the new bridge, if I edit the qemu conf file for the VM, the port is added and the VM starts.


  • LAYER 8 Netgate

    Sounds like you should probably move to a proxmox forum.


Log in to reply