What is the VMM Network selection for a LAN and WAN NIC
-
@viragomann Yes, I've updated the bridge connection via ip link commands and have the NIC0-br0 and NIC1-br1.
However, this Ubuntu 22.04 machine 192.168.1.120 and the guest vm pfSense 192.168.1.1 won't ping each other?ubuntu@ubuntu:/etc/netplan$ ip -c a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br1 state UP group default qlen 1000 link/ether 1c:61:b4:6d:38:4f brd ff:ff:ff:ff:ff:ff inet6 fe80::31ca:9227:dcb3:d09e/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000 link/ether a8:a1:59:6e:1f:8b brd ff:ff:ff:ff:ff:ff 4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 52:54:00:88:b4:b4 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 7: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fe:54:00:33:3c:4b brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fe33:3c4b/64 scope link valid_lft forever preferred_lft forever 8: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fe:54:00:50:81:3f brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fe50:813f/64 scope link valid_lft forever preferred_lft forever 9: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 0a:d4:78:1f:cd:41 brd ff:ff:ff:ff:ff:ff inet 100.76.25.213/10 brd 100.127.255.255 scope global dynamic noprefixroute br0 valid_lft 294sec preferred_lft 294sec inet6 2406:2d40:4100:8fb2:19c5:376e:1317:8ae1/64 scope global temporary dynamic valid_lft 197sec preferred_lft 47sec inet6 2406:2d40:4100:8fb2:8d4:78ff:fe1f:cd41/64 scope global dynamic mngtmpaddr valid_lft 197sec preferred_lft 47sec inet6 fe80::8d4:78ff:fe1f:cd41/64 scope link valid_lft forever preferred_lft forever 10: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether d2:da:46:a2:b2:3e brd ff:ff:ff:ff:ff:ff inet 192.168.1.120/24 brd 192.168.1.255 scope global noprefixroute br1 valid_lft forever preferred_lft forever inet6 fe80::d0da:46ff:fea2:b23e/64 scope link valid_lft forever preferred_lft forever ubuntu@ubuntu:/etc/netplan$ bridge link 2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br1 state forwarding priority 32 cost 4 3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100 ubuntu@ubuntu:/etc/netplan$ cat 01-network-manager-all.yaml # Let NetworkManager manage all devices on this system network: version: 2 renderer: NetworkManager ethernets: enp2s0: dhcp4: true enp3s0: dhcp4: true bridges: br0: dhcp4: true interfaces: - enp3s0 br1: dhcp4: false addresses: [192.168.1.120/24] gateway4: 192.168.1.1 nameservers: addresses: [8.8.8.8,8.8.8.4] interfaces: - enp2s0 ubuntu@ubuntu:/etc/netplan$ ip r default via 100.64.0.1 dev br0 proto dhcp metric 425 default via 192.168.1.1 dev br1 proto static metric 20426 34.120.255.244 dev br0 proto dhcp scope link metric 425 100.64.0.0/10 dev br0 proto kernel scope link src 100.76.25.213 metric 425 169.254.0.0/16 dev virbr0 scope link metric 1000 linkdown 192.168.1.0/24 dev br1 proto kernel scope link src 192.168.1.120 metric 426 192.168.100.1 dev br0 proto dhcp scope link metric 425 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown -
@eiger3970-0
The "ip a" output doesn't look plausible to me. If a single NIC is slave in a bridge, I'd expect that the bridge takes over the MAC from the NIC.Also there should not be any IP settings on the NIC port. However, your enp2s0 shows IPv6 settings.
Also I cannot see that any of your virtual interfaces is connected to one of the bridges. So it's expected that pinging doesn't work.
Did you set the bridge name in VMM? -
@viragomann Maybe I'm missing the connection, but not sure what to do? I connect NIC0-br0 and NIC1-br1. Maybe I need something like NIC0-br0-vtnet0 and NIC1-br1-vtnet1?
I create bridges for the VMM, in this order.
1: Ubuntu 22.04 Host machine:
sudo ip link set dev enp3s0 master br0
sudo ip link set dev enp2s0 master br12: Ubuntu 22.04 Host machine:
/etc/netplan/01-network-manager-all.yaml# Let NetworkManager manage all devices on this system network: version: 2 renderer: NetworkManager ethernets: enp2s0: dhcp4: true enp3s0: dhcp4: true bridges: br0: dhcp4: true interfaces: - enp3s0 br1: dhcp4: false addresses: [192.168.1.120/24] gateway4: 192.168.1.1 nameservers: addresses: [8.8.8.8,8.8.8.4] interfaces: - enp2s03: Virtual Machine Manager 4.0.0:
Virtual Network Interface > Network Source: Bridge device... > Device name: br0.
Add Hardware > Virtual Network Interface > Network Source: Bridge device... > Device name: br1. -
@eiger3970-0 said in What is the VMM Network selection for a LAN and WAN NIC:
3: Virtual Machine Manager 4.0.0:
Virtual Network Interface > Network Source: Bridge device... > Device name: br0.
Add Hardware > Virtual Network Interface > Network Source: Bridge device... > Device name: br1.That's how I did it as well. But running "ip a", my virtual NICs show the bridges they're connected to:
However, as already mentioned, I don't use the Network Manager.
Do you use the host also for other purposes? If it's only for virtualization I'd consider to install Proxmox or alike on it.
-
@viragomann I just use the host machine as a personal desktop.
I previously ran Proxmox which ran nicely, however had to run Ubuntu as a vm.
My hope is to run Ubuntu with better performance not being a vm any longer. -
@eiger3970-0 I've reinstalled the Ubuntu 22.04, VMM 4.0.0 and VM pfSense.
Ubuntu 192.168.1.120 pings the Internet (very slow connection though), vm router and LAN devices.
VM router 192.168.1.1 pings Ubuntu and LAN devices, however not the Internet.
This tells me Internet is running via the Ubuntu 22.04 iptables, rather than via the vm router.
Somehow I think I'm missing bridging the WAN interface NIC0-br0-vm router?
Here's my latest configuration:root@ubuntu:/etc/netplan# cat 50-cloud-init.yaml network: version: 2 renderer: networkd ethernets: enp2s0: dhcp4: false dhcp6: false enp3s0: dhcp4: false dhcp6: false bridges: br0: interfaces: [enp3s0] dhcp4: true dhcp6: true br1: interfaces: [enp2s0] addresses: [192.168.1.120/24] # gateway4 is deprecated, use routes instead. routes: - to: default via: 192.168.1.1 metric: 100 on-link: true mtu: 1500 nameservers: addresses: [8.8.8.8] parameters: stp: true forward-delay: 4 dhcp4: no dhcp6: noroot@ubuntu:/etc/netplan# ip -c a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br1 state UP group default qlen 1000 link/ether 1c:61:b4:6d:38:4f brd ff:ff:ff:ff:ff:ff 3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000 link/ether a8:a1:59:6e:1f:8b brd ff:ff:ff:ff:ff:ff 4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000 link/ether 52:54:00:a9:81:30 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 8: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN group default qlen 1000 link/ether fe:54:00:d4:d4:20 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fed4:d420/64 scope link valid_lft forever preferred_lft forever 9: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether e2:73:dd:ac:ba:e0 brd ff:ff:ff:ff:ff:ff inet 100.76.25.213/10 metric 100 brd 100.127.255.255 scope global dynamic br0 valid_lft 187sec preferred_lft 187sec inet6 2406:2d40:4100:8fb2:e073:ddff:feac:bae0/64 scope global dynamic mngtmpaddr noprefixroute valid_lft 233sec preferred_lft 83sec inet6 fe80::e073:ddff:feac:bae0/64 scope link valid_lft forever preferred_lft forever 10: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether ce:08:2f:30:4a:71 brd ff:ff:ff:ff:ff:ff inet 192.168.1.120/24 brd 192.168.1.255 scope global br1 valid_lft forever preferred_lft forever inet6 fe80::cc08:2fff:fe30:4a71/64 scope link valid_lft forever preferred_lft forever 11: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1 state UNKNOWN group default qlen 1000 link/ether fe:54:00:95:1e:e5 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fe95:1ee5/64 scope link valid_lft forever preferred_lft foreverroot@ubuntu:/etc/netplan# ip r default via 192.168.1.1 dev br1 proto static metric 100 onlink default via 100.64.0.1 dev br0 proto dhcp src 100.76.25.213 metric 100 1.1.1.1 via 100.64.0.1 dev br0 proto dhcp src 100.76.25.213 metric 100 8.8.8.8 via 100.64.0.1 dev br0 proto dhcp src 100.76.25.213 metric 100 34.120.255.244 dev br0 proto dhcp scope link src 100.76.25.213 metric 100 100.64.0.0/10 dev br0 proto kernel scope link src 100.76.25.213 metric 100 100.64.0.1 dev br0 proto dhcp scope link src 100.76.25.213 metric 100 169.254.0.0/16 dev virbr0 scope link metric 1000 linkdown 192.168.1.0/24 dev br1 proto kernel scope link src 192.168.1.120 192.168.100.1 dev br0 proto dhcp scope link src 100.76.25.213 metric 100 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdownroot@ubuntu:/etc/netplan# bridge link 2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br1 state forwarding priority 32 cost 4 3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 4 8: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100 11: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br1 state forwarding priority 32 cost 100 -
@eiger3970-0 said in What is the VMM Network selection for a LAN and WAN NIC:
buntu 192.168.1.120 pings the Internet (very slow connection though)
It has two routes for upstream connection. The one via the pfSense VM might fail.
VM router 192.168.1.1 pings Ubuntu and LAN devices, however not the Internet.
What is its WAN configuration and default gateway?
-
@viragomann
What are the 2 routes for upstream connection, I don't clearly see them?I think these are the LAN and WAN details:
Ubuntu 22.04 Desktop > Browser > 192.168.1.1 > Enter > pfSense > Lobby > Dashboard > Interfaces > LAN 10Gbase-T <full-duplex> 192.168.1.1 track6.
Ubuntu 22.04 Desktop > Browser > 192.168.1.1 > Enter > pfSense > Lobby > Dashboard > Interfaces > WAN 10Gbase-T <full-duplex> 100.76.25.213 2406.2d40:4100:8fb2:5054:ff:fed4:d420I think these are the Gateways details:
Ubuntu 22.04 Desktop > Browser > 192.168.1.1 > Enter > pfSense > Lobby > Dashboard > Gateways > WAN_DHCP6 fe80::200:5eff:fe00:101
Ubuntu 22.04 Desktop > Browser > 192.168.1.1 > Enter > pfSense > Lobby > Dashboard > Gateways > WAN_DHCP 100.64.0.1.These are all default settings that appeared in pfSense after I configured Ubuntu 22.04 Desktop's Netplan and turned off NetworkManager as per this tutorial.
-
default via 192.168.1.1 dev br1 proto static metric 100 onlink
default via 100.64.0.1 dev br0 proto dhcp src 100.76.25.213 metric 100You have the first one set manually. The latter is due to the DHCP client enabled on br0. So the bridge gets a config from the ISP router presumably.
If you want to go out via pfSense disable the DHCP client on br0 and enable it on pfSense WAN, assuming this is connected to br0.
-
@viragomann Thank you for the clarification.
This is what I've been trying to do, to have traffic via the vm router.
Any suggestions, as if I disable the br0's DHCP, it must need some address for a static IP?
How do I check the vm router is connected to br0?
How do I connect the vm router to br0?
How do I check the vm router's WAN is on DHCP? The vm router's Interfaces: [WAN] IPv4 Configuration Type, does say it's on DHCP, so this should be ok? -
@eiger3970-0 I've configured Ubuntu 22.04 to use renderer networkd rather than NetworkManager.
I can now route the Ubuntu host via the vm router for Internet and LAN access.However, only some Internet loads and ping 8.8.8.8 has 100% packet loss?
I think the pfSense configuration needs some details I'm missing.
Any paid support to finalise my configuration please?
I've set LAN and WAN to static IPs, but will probably fail once the ISP changes the public IP. -
@eiger3970-0 said in What is the VMM Network selection for a LAN and WAN NIC:
This updated diagram might help for clarity?
That diagram looks wrong to me. Specifically
Wan side would normally be
ISP <-> Modem (Bridged router) <-> Nic0/eth0/snp3s0 <-> br0 <-> pfSenseVM (and no other VM or hypervisor)Lan side would normally be
pfSenseVM <-> br1 <-> other VMs, Hypervisor, and Nic1/eth1/enp2s0 <-> external switch <-> Wired Lan devices, Wifi APSo in your diagram I don't like br0 connection to VM x and VM y
Nor do I like the non specific connection of the Hypervisor to the pfsense router
- The hypervisor network access should be explicitly via the Lan / br1. In Proxmox that is set by assigning an IP address to the Vbr1. Less sure how to do it directly in Ubuntu 22.04.
- The hypervisor also has VM access via the VM console and Qemu but that is separate to network access
-
@Patch thanks, your reply seems right.
The setup is now working, but I haven’t got my head 100% around it yet.
I’m figuring out how to make a clear and correct topology for easy understanding and replication. -
@eiger3970-0 I have created a new topology map.
However, I am unclear the the packet routes?
As per ip route, line 2 is the default route of this host machine Ubuntu 22.04 LTS Desktop.
100.64.0.1 came from a DHCP, but where is the DHCP from, the VM router or the ISP?ubuntu@ubuntu:/etc/netplan$ ip -c a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master local-br1.1 state UP group default qlen 1000 link/ether 46:9e:01:27:db:0d brd ff:ff:ff:ff:ff:ff permaddr 1c:61:b4:6d:38:4f 3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether a8:a1:59:6e:1f:8b brd ff:ff:ff:ff:ff:ff inet 100.76.25.213/10 brd 100.127.255.255 scope global dynamic noprefixroute enp3s0 valid_lft 168sec preferred_lft 168sec inet6 2406:2d40:4100:8fb2:7b54:ac94:277b:140f/64 scope global temporary deprecated dynamic valid_lft 234sec preferred_lft 0sec inet6 2406:2d40:4100:8fb2:f995:e028:b158:4865/64 scope global dynamic mngtmpaddr noprefixroute valid_lft 234sec preferred_lft 84sec inet6 fe80::8cc1:c5d0:5e3e:28b1/64 scope link noprefixroute valid_lft forever preferred_lft forever 4: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:94:0d:bb brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever 5: local-br1.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 2e:f6:bf:42:0c:f8 brd ff:ff:ff:ff:ff:ff inet 192.168.1.120/24 brd 192.168.1.255 scope global noprefixroute local-br1.1 valid_lft forever preferred_lft forever 6: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master local-br1.1 state UNKNOWN group default qlen 1000 link/ether fe:54:00:3d:53:41 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fe3d:5341/64 scope link valid_lft forever preferred_lft forever 7: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master virbr0 state UNKNOWN group default qlen 1000 link/ether fe:54:00:a3:f6:c4 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fea3:f6c4/64 scope link valid_lft forever preferred_lft forever 16: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master local-br1.1 state UNKNOWN group default qlen 1000 link/ether fe:54:00:8d:31:06 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fe8d:3106/64 scope link valid_lft forever preferred_lft forever 17: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master local-br1.1 state UNKNOWN group default qlen 1000 link/ether fe:54:00:f7:f9:e5 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fef7:f9e5/64 scope link valid_lft forever preferred_lft forever 18: vnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master local-br1.1 state UNKNOWN group default qlen 1000 link/ether fe:54:00:b6:35:37 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:feb6:3537/64 scope link valid_lft forever preferred_lft foreverubuntu@ubuntu:/etc/netplan$ arp -n Address HWtype HWaddress Flags Mask Iface 192.168.1.160 ether 3c:2a:f4:60:b5:7b C local-br1.1 192.168.1.1 ether 52:54:00:3d:53:41 C local-br1.1 192.168.122.149 ether 52:54:00:a3:f6:c4 C virbr0 192.168.1.102 ether 52:54:00:8d:31:06 C local-br1.1 192.168.1.100 ether 1c:ca:e3:77:c5:53 C local-br1.1 100.64.0.1 ether 00:00:5e:00:01:01 C enp3s0ubuntu@ubuntu:/etc/netplan$ ip route default via 100.64.0.1 dev enp3s0 proto dhcp metric 102 34.120.255.244 dev enp3s0 proto dhcp scope link metric 102 100.64.0.0/10 dev enp3s0 proto kernel scope link src 100.76.25.213 metric 102 169.254.0.0/16 dev virbr0 scope link metric 1000 192.168.1.0/24 dev local-br1.1 proto kernel scope link src 192.168.1.120 metric 425 192.168.100.1 dev enp3s0 proto dhcp scope link metric 102 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 -
@eiger3970-0 updated version of topology map.
ubuntu@ubuntu:/etc/netplan$ route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default _gateway 0.0.0.0 UG 102 0 0 enp3s0 244.255.120.34. 0.0.0.0 255.255.255.255 UH 102 0 0 enp3s0 100.64.0.0 0.0.0.0 255.192.0.0 U 102 0 0 enp3s0 link-local 0.0.0.0 255.255.0.0 U 1000 0 0 virbr0 192.168.1.0 0.0.0.0 255.255.255.0 U 425 0 0 local-br1.1 192.168.100.1 0.0.0.0 255.255.255.255 UH 102 0 0 enp3s0 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0
-
This post is deleted! -
@eiger3970-0 this is as far as I can create the topology map
-
@eiger3970-0 A bridge is the virtual equivalent of a multi-port switch.
You will need two virtual switches / bridges
-
Wan bridge connects the physical NIC you want to use for WAN connection your VM routers WAN virtual NIC. I think you a meaning to use vbr0 for this but have not connected to your router VM's Wan virtual NIC.
-
Lan bridge connects the phyiscal NIC you want to use to connect to your physical 24 port switch externally. It also connects your VM which need LAN access (KVM X, Y, Z) as well as your hypervisor network access and your virtual routers Lan virtual NIC. I think you have sort of used vbr1 for this.
-
-
@Patch Thanks, this is working now, just need to sort out some port forwards.