Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    What is the VMM Network selection for a LAN and WAN NIC

    Virtualization
    3
    33
    4.1k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • E
      eiger3970 0
      last edited by

      I am building pretty much a similar setup to Proxmox which is a host running a VM router for traffic for the LAN and WAN. As Proxmox uses KVM, I thought it should work direct from my machine's KVM?

      My setup:
      1 bridged router.
      1 machine running Debian based OS Ubuntu 22.04.
      2 physical LAN and WAN wired NICs.
      1 host/hypervisor/VMM KVM/Qemu/Virtual Machine Manager 4.0.0.
      1 guest/VM router pfSense 2.6.0.

      I need the LAN (including the host machine) and WAN to route traffic via the VM router only.
      The machine recognises the 2 NICs.
      I think the host needs 1 or 2 bridges to the machine's physical LAN and WAN NICs?
      Here's my network topology which may not be the right setup, attached. alt text

      Configuring the network has a few options like
      GUI: Virtual Manager.
      TUI: Network scripts, Nmcli tool or Virsh command.
      I tried the host GUI: alt textNew VM > Create a new virtual machine > Step 5 of 5 > Ready to begin the installation > Network selection > Bridge device… > Device name: enp3s0 > Finish > Unable to complete install: ‘Unable to add bridge enp3s0 port vnet0: Operation not supported’.
      Details:

      Unable to complete install: 'Unable to add bridge enp3s0 port vnet0: Operation not supported'

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 72, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/createvm.py", line 2008, in _do_async_install
    installer.start_install(guest, meter=meter)
  File "/usr/share/virt-manager/virtinst/install/installer.py", line 695, in start_install
    domain = self._create_guest(
  File "/usr/share/virt-manager/virtinst/install/installer.py", line 637, in _create_guest
    domain = self.conn.createXML(initial_xml or final_xml, 0)
  File "/usr/lib/python3/dist-packages/libvirt.py", line 4400, in createXML
    raise libvirtError('virDomainCreateXML() failed')
libvirt.libvirtError: Unable to add bridge enp3s0 port vnet0: Operation not supported

      The host has an option for Macvtap device, but I don't know what this is?

      V 1 Reply Last reply Reply Quote 0
      • V
        viragomann @eiger3970 0
        last edited by

        @eiger3970-0 said in What is the VMM Network selection for a LAN and WAN NIC:

        I need the LAN (including the host machine) and WAN to route traffic via the VM router only.
        The machine recognises the 2 NICs.> I think the host needs 1 or 2 bridges to the machine's physical LAN and WAN NICs?

        Yes, best to configure two Linux bridges and connect each to one of the network ports. So that you have a lan-bridge and a wan-bridge.
        Assign on IP to the LAN bridge for the host. So you can access the host from a devices connected to the LAN port, but route upstream traffic through pfSense.

        VM > Create a new virtual machine > Step 5 of 5 > Ready to begin the installation > Network selection > Bridge device… > Device name: enp3s0 > Finish > Unable to complete install: ‘Unable to add bridge enp3s0 port vnet0: Operation not supported’.

        This is the network port. You have to state a bridge name here.

        You have to create these bridges before in the Linux shell or in the network management GUI if any. But it probably cannot be done in VMM.
        You can check the options you have in VMM by double clicking on "QEMU/KVM".
        657f38f5-34d7-4098-b938-9b96e6dbe6b7-grafik.png

        E 1 Reply Last reply Reply Quote 0
        • E
          eiger3970 0 @viragomann
          last edited by eiger3970 0

          @viragomann Ok, I've manually added bridges:

          /etc/netplan$ cat 01-network-manager-all.yaml
# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager
  ethernets:
    enp2s0:
      dhcp4: no
    enp3s0:
      dhcp4: no
  bridges:
    br0:
      dhcp4: yes
      interfaces:
        - enp3s0
    br1:
      dhcp4: yes
      interfaces:
        - enp2s0

          It seems VMM 4.0.0 has overridden my bridges br0 and br1 by adding its own bridge virbr0.

          ubuntu@ubuntu:/etc/netplan$ nmcli connection show --active 
NAME            UUID                                  TYPE      DEVICE 
netplan-br0     00679506-5c05-3c3d-bdfe-474849762078  bridge    br0    
virbr0          41fc1185-7dab-4502-a514-6fbd9f6294bd  bridge    virbr0 
vnet0           ea1f400d-8883-469f-9fbf-fdea86ec6bb2  tun       vnet0  
netplan-enp3s0  6effa1b1-280b-3785-9b52-c723b445fb3e  ethernet  enp3s0 
ubuntu@ubuntu:/etc/netplan$ brctl show
bridge name	bridge id		STP enabled	interfaces
br0		8000.0ad4781fcd41	yes		enp3s0
							vnet0
br1		8000.d2da46a2b23e	yes		enp2s0
virbr0		8000.52540088b4b4	yes

          So, I'm confused here as to what interface is connected to what bridge.

          Then, to configure pfSense I need to access 192.168.1.1, however I am unsure how to set what interface to reach 192.168.1.1.

          $ ip -c a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br1 state UP group default qlen 1000
    link/ether 1c:61:b4:6d:38:4f brd ff:ff:ff:ff:ff:ff
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether a8:a1:59:6e:1f:8b brd ff:ff:ff:ff:ff:ff
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:88:b4:b4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0a:d4:78:1f:cd:41 brd ff:ff:ff:ff:ff:ff
    inet 100.76.25.213/10 brd 100.127.255.255 scope global dynamic noprefixroute br0
       valid_lft 172sec preferred_lft 172sec
    inet6 2406:2d40:4100:8fb2:ef95:6dbe:83e8:15b6/64 scope global temporary deprecated dynamic 
       valid_lft 241sec preferred_lft 0sec
    inet6 2406:2d40:4100:8fb2:3f8f:8058:a24e:90d7/64 scope global temporary deprecated dynamic 
       valid_lft 241sec preferred_lft 0sec
    inet6 2406:2d40:4100:8fb2:8d4:78ff:fe1f:cd41/64 scope global dynamic mngtmpaddr 
       valid_lft 241sec preferred_lft 91sec
    inet6 fe80::8d4:78ff:fe1f:cd41/64 scope link 
       valid_lft forever preferred_lft forever
627: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:9f:47:34 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe9f:4734/64 scope link 
       valid_lft forever preferred_lft forever
651: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fe:54:00:a2:19:8b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fea2:198b/64 scope link 
       valid_lft forever preferred_lft forever
2764: br1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000

          In pfSense TUI, I've assigned the interfaces:
          LAN (vtnet1) -> v4: 192.168.1.1/24
          WAN (vtnet0) -> v4/DHCP4: 0.0.0.0/8
          v6/DHCP6: 2406:2d40:4100:8fb2:5054:ff:fe9f:4734/64

          V 1 Reply Last reply Reply Quote 0
          • V
            viragomann @eiger3970 0
            last edited by

            @eiger3970-0 said in What is the VMM Network selection for a LAN and WAN NIC:

            It seems VMM 4.0.0 has overridden my bridges br0 and br1 by adding its own bridge virbr0

            This is a virtual network, created by VMM, not a real bridge.

            So, I'm confused here as to what interface is connected to what bridge.

            Use the bridge command to show all connections:

            bridge link show

            Then, to configure pfSense I need to access 192.168.1.1, however I am unsure how to set what interface to reach 192.168.1.1.

            In VMM connect the pfSense VM to the desired bridge. Select "bridge device" as network source and enter the name of the bridge (e.g. br0).

            1 Reply Last reply Reply Quote 0
            • E
              eiger3970 0
              last edited by

              It appears when I run the Guest/VM/router/firewall pfSense, the Hypervisor/host/vmm KVM/Qemu changes my bridge settings by replacing br1 with virbr0?
              From my understanding, this is or should be the network architecture/topology:
              alt text

              I set the host Ubuntu 22.04 network manager with the following settings:

              ubuntu@ubuntu:/etc/netplan$ cat 01-network-manager-all.yaml
# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager
  ethernets:
    enp2s0:
      dhcp4: no
    enp3s0:
      dhcp4: no
  bridges:
    br0:
      dhcp4: yes
      interfaces:
        - enp3s0
    br1:
      dhcp4: yes
      interfaces:
        - enp2s0

              With VM pfSense not running, the Ubuntu 22.04 Terminal outputs are:

              ubuntu@ubuntu:/etc/netplan$ ip -c a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br1 state UP group default qlen 1000
    link/ether 1c:61:b4:6d:38:4f brd ff:ff:ff:ff:ff:ff
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether a8:a1:59:6e:1f:8b brd ff:ff:ff:ff:ff:ff
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:88:b4:b4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0a:d4:78:1f:cd:41 brd ff:ff:ff:ff:ff:ff
    inet 100.76.25.213/10 brd 100.127.255.255 scope global dynamic noprefixroute br0
       valid_lft 240sec preferred_lft 240sec
    inet6 2406:2d40:4100:8fb2:b72a:cfe1:5980:954b/64 scope global temporary deprecated dynamic 
       valid_lft 254sec preferred_lft 0sec
    inet6 2406:2d40:4100:8fb2:8d4:78ff:fe1f:cd41/64 scope global dynamic mngtmpaddr 
       valid_lft 254sec preferred_lft 104sec
    inet6 fe80::8d4:78ff:fe1f:cd41/64 scope link 
       valid_lft forever preferred_lft forever
6795: br1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether d2:da:46:a2:b2:3e brd ff:ff:ff:ff:ff:ff
ubuntu@ubuntu:/etc/netplan$ bridge link show
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br1 state forwarding priority 32 cost 100 
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100 
ubuntu@ubuntu:/etc/netplan$ brctl show
bridge name	bridge id		STP enabled	interfaces
br0		8000.0ad4781fcd41	yes		enp3s0
br1		8000.d2da46a2b23e	yes		enp2s0
virbr0		8000.52540088b4b4	yes

              With VM pfSense running, the Ubuntu 22.04 Terminal outputs are:

              ubuntu@ubuntu:/etc/netplan$ ip -c a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 1c:61:b4:6d:38:4f brd ff:ff:ff:ff:ff:ff
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether a8:a1:59:6e:1f:8b brd ff:ff:ff:ff:ff:ff
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:88:b4:b4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0a:d4:78:1f:cd:41 brd ff:ff:ff:ff:ff:ff
    inet 100.76.25.213/10 brd 100.127.255.255 scope global dynamic noprefixroute br0
       valid_lft 289sec preferred_lft 289sec
    inet6 2406:2d40:4100:8fb2:b72a:cfe1:5980:954b/64 scope global temporary deprecated dynamic 
       valid_lft 225sec preferred_lft 0sec
    inet6 2406:2d40:4100:8fb2:8d4:78ff:fe1f:cd41/64 scope global dynamic mngtmpaddr 
       valid_lft 225sec preferred_lft 75sec
    inet6 fe80::8d4:78ff:fe1f:cd41/64 scope link 
       valid_lft forever preferred_lft forever
627: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:9f:47:34 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe9f:4734/64 scope link 
       valid_lft forever preferred_lft forever
651: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fe:54:00:a2:19:8b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fea2:198b/64 scope link 
       valid_lft forever preferred_lft forever
ubuntu@ubuntu:/etc/netplan$ bridge link show
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100 
627: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100 
ubuntu@ubuntu:/etc/netplan$ brctl show
bridge name	bridge id		STP enabled	interfaces
br0		8000.0ad4781fcd41	yes		enp3s0
							vnet0
virbr0		8000.52540088b4b4	yes
              E 1 Reply Last reply Reply Quote 0
              • E
                eiger3970 0 @eiger3970 0
                last edited by eiger3970 0

                @eiger3970-0 This updated diagram might help for clarity?
                alt text

                E P 2 Replies Last reply Reply Quote 0
                • E
                  eiger3970 0 @eiger3970 0
                  last edited by

                  @eiger3970-0
                  I updated the host Ubuntu's Network Manager, but no LAN IP 192.168.1.120 appears and no connection to the guest pfSense router on 192.168.1.1?

                  ubuntu@ubuntu:/etc/netplan$ cat 01-network-manager-all.yaml
# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager
  ethernets:
    enp2s0:
      dhcp4: no
      addresses: [192.168.1.120/24]
      gateway4: 192.168.1.1
      nameservers:
        addresses: [8.8.8.8,8.8.8.4]
    enp3s0:
      dhcp4: no
  bridges:
    br0:
      dhcp4: yes
      interfaces:
        - enp3s0
    br1:
      dhcp4: yes
      interfaces:
        - enp2s0


                  ubuntu@ubuntu:/etc/netplan$ ip -c a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 1c:61:b4:6d:38:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::31ca:9227:dcb3:d09e/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether a8:a1:59:6e:1f:8b brd ff:ff:ff:ff:ff:ff
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:88:b4:b4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0a:d4:78:1f:cd:41 brd ff:ff:ff:ff:ff:ff
    inet 100.76.25.213/10 brd 100.127.255.255 scope global dynamic noprefixroute br0
       valid_lft 217sec preferred_lft 217sec
    inet6 2406:2d40:4100:8fb2:c261:34a2:7275:ff6e/64 scope global temporary dynamic 
       valid_lft 201sec preferred_lft 51sec
    inet6 2406:2d40:4100:8fb2:8d4:78ff:fe1f:cd41/64 scope global dynamic mngtmpaddr 
       valid_lft 201sec preferred_lft 51sec
    inet6 fe80::8d4:78ff:fe1f:cd41/64 scope link 
       valid_lft forever preferred_lft forever
8: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:c9:2f:af brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fec9:2faf/64 scope link 
       valid_lft forever preferred_lft forever
9: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fe:54:00:ad:30:31 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fead:3031/64 scope link 
       valid_lft forever preferred_lft forever
499: br1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether d2:da:46:a2:b2:3e brd ff:ff:ff:ff:ff:ff
                  V 1 Reply Last reply Reply Quote 0
                  • V
                    viragomann @eiger3970 0
                    last edited by

                    @eiger3970-0
                    Whats the reason for setting an IP on enp2s0 now?
                    You have added it to br1 as member, which you have enabled the DHCP client on. That might conflict.
                    If you want to have a static IP set it on the bridge and remove from enp2s0.

                    In your last ip command output enp2s0 is not shown as bridge member anymore. However, it has an IPv6, what I don't expect according to the settings. So it seems that there is something wrong with the br1/enp2s0.

                    Also I wondering what's your reason to have the DHCP client active on both bridges.

                    Is there no other network configuration tool available on Ubuntu than Network Manager?
                    I use it only on the desktop for my purposes. But on a server I use to configure the network with wicked. It's more reliable in my opinion.

                    E 1 Reply Last reply Reply Quote 0
                    • E
                      eiger3970 0 @viragomann
                      last edited by eiger3970 0

                      @viragomann Before a host reboot:

                      ubuntu@ubuntu:/etc/netplan$ cat 01-network-manager-all.yaml
# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager
  ethernets:
    enp2s0:
      dhcp4: yes 
    enp3s0:
      dhcp4: yes
  bridges:
    br0:
      dhcp4: yes
      interfaces:
        - enp3s0
        - vtnet0
    br1:
      dhcp4: no
      addresses: [192.168.1.120/24]
      gateway4: 192.168.1.1
      nameservers:
        addresses: [8.8.8.8,8.8.8.4]
      interfaces:
        - enp2s0
        - vtnet0
ubuntu@ubuntu:/etc/netplan$ sudo netplan try

** (process:22120): WARNING **: 14:50:09.654: `gateway4` has been deprecated, use default routes instead.
See the 'Default routes' section of the documentation for more details.
/etc/netplan/01-network-manager-all.yaml:24:11: Error in network definition: br1: interface 'vtnet0' is not defined
        - vtnet0
          ^
ubuntu@ubuntu:/etc/netplan$ ip -c a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 1c:61:b4:6d:38:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::31ca:9227:dcb3:d09e/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether a8:a1:59:6e:1f:8b brd ff:ff:ff:ff:ff:ff
    inet 100.76.25.213/10 brd 100.127.255.255 scope global dynamic noprefixroute enp3s0
       valid_lft 206sec preferred_lft 206sec
    inet6 2406:2d40:4100:8fb2:a33c:3599:7805:669/64 scope global temporary dynamic 
       valid_lft 240sec preferred_lft 90sec
    inet6 2406:2d40:4100:8fb2:14a9:5501:ce7c:4421/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 240sec preferred_lft 90sec
    inet6 fe80::5be3:7f2a:2da2:5285/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:88:b4:b4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever

                      After reboot, no Internet.

                      ubuntu@ubuntu:~$ ip -c a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 1c:61:b4:6d:38:4f brd ff:ff:ff:ff:ff:ff
3: enp3s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:a1:59:6e:1f:8b brd ff:ff:ff:ff:ff:ff
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:88:b4:b4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever

                      No Internet as I think the Host is looking for the Guest pfSense router.
                      However, this is the next issue the VMM won't add 1 or 2 bridges.
                      Starting the Guest Error starting domain: Cannot get interface MTU on 'br0': No such device.

                      V 1 Reply Last reply Reply Quote 0
                      • V
                        viragomann @eiger3970 0
                        last edited by

                        @eiger3970-0 said in What is the VMM Network selection for a LAN and WAN NIC:

                        However, this is the next issue the VMM won't add 1 or 2 bridges.
                        Starting the Guest Error starting domain: Cannot get interface MTU on 'br0': No such device.

                        No surprise at all. There is no br0 configured.

                        Obviously you have no proper bridge configuration now. None of your NICs is assigned to the only one bridge which is called virbr0.

                        E 2 Replies Last reply Reply Quote 1
                        • E
                          eiger3970 0 @viragomann
                          last edited by

                          This post is deleted!
                          1 Reply Last reply Reply Quote 0
                          • E
                            eiger3970 0 @viragomann
                            last edited by

                            @viragomann I've realised the bridges I create NIC0-br0 and NIC1-br1, disappear after rebooting the machine?
                            However virbr0 appears (probably from KVM) and stays after a reboot.

                            The main issue now seems to be the Ubuntu 22.04 machine can't ping the vm router on 192.168.1.1.
                            I can't quite figure out the host machine's static ip, even though I thought I set it in /etc/netplan/01-network-manager-all.yaml.

                            E 1 Reply Last reply Reply Quote 0
                            • E
                              eiger3970 0 @eiger3970 0
                              last edited by eiger3970 0

                              @eiger3970-0
                              After a few Ubuntu 22.04 reboots, the br0 and br1 appear permanent.
                              However, whilst br0 connectes to enp3s0, br1 won't stay connected to enp2s0?
                              br0 automatically connects to vnet1, after I start up the vm.
                              Confusing?

                              ubuntu@ubuntu:/etc/netplan$ ip -c a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 1c:61:b4:6d:38:4f brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.120/24 brd 192.168.1.255 scope global noprefixroute enp2s0
       valid_lft forever preferred_lft forever
    inet6 fe80::31ca:9227:dcb3:d09e/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether a8:a1:59:6e:1f:8b brd ff:ff:ff:ff:ff:ff
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:88:b4:b4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
5: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d2:da:46:a2:b2:3e brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.120/24 brd 192.168.1.255 scope global noprefixroute br1
       valid_lft forever preferred_lft forever
    inet6 fe80::d0da:46ff:fea2:b23e/64 scope link 
       valid_lft forever preferred_lft forever
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0a:d4:78:1f:cd:41 brd ff:ff:ff:ff:ff:ff
    inet 100.76.25.213/10 brd 100.127.255.255 scope global dynamic noprefixroute br0
       valid_lft 267sec preferred_lft 267sec
    inet6 2406:2d40:4100:8fb2:20eb:c8df:ba1e:9e3e/64 scope global temporary dynamic 
       valid_lft 202sec preferred_lft 52sec
    inet6 2406:2d40:4100:8fb2:8d4:78ff:fe1f:cd41/64 scope global dynamic mngtmpaddr 
       valid_lft 202sec preferred_lft 52sec
    inet6 fe80::8d4:78ff:fe1f:cd41/64 scope link 
       valid_lft forever preferred_lft forever
7: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:33:3c:4b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe33:3c4b/64 scope link 
       valid_lft forever preferred_lft forever
8: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:50:81:3f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe50:813f/64 scope link 
       valid_lft forever preferred_lft forever
ubuntu@ubuntu:/etc/netplan$ bridge link
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100 
7: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100 
8: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br1 state forwarding priority 32 cost 100 
ubuntu@ubuntu:/etc/netplan$ ubuntu@ubuntu:/etc/netplan$ cat 01-network-manager-all.yaml
# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager
  ethernets:
    enp2s0:
      dhcp4: true 
    enp3s0:
      dhcp4: true
  bridges:
    br0:
      dhcp4: true
      interfaces:
        - enp3s0
    br1:
      dhcp4: false
      addresses: [192.168.1.120/24]
      gateway4: 192.168.1.1
      nameservers:
        addresses: [8.8.8.8,8.8.8.4]
      interfaces:
        - enp2s0
                              V 1 Reply Last reply Reply Quote 0
                              • V
                                viragomann @eiger3970 0
                                last edited by

                                @eiger3970-0
                                I'd kick the Network Manager away and do it with ifup / ifdown scripts.
                                This is a server, the settings need to be done only once.

                                I only use NM on the desktop and notebook, because you can easily make a VPN connection with it.

                                E 1 Reply Last reply Reply Quote 1
                                • E
                                  eiger3970 0 @viragomann
                                  last edited by

                                  @viragomann Yes, I've updated the bridge connection via ip link commands and have the NIC0-br0 and NIC1-br1.
                                  However, this Ubuntu 22.04 machine 192.168.1.120 and the guest vm pfSense 192.168.1.1 won't ping each other?

                                  ubuntu@ubuntu:/etc/netplan$ ip -c a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br1 state UP group default qlen 1000
    link/ether 1c:61:b4:6d:38:4f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::31ca:9227:dcb3:d09e/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether a8:a1:59:6e:1f:8b brd ff:ff:ff:ff:ff:ff
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:88:b4:b4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
7: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fe:54:00:33:3c:4b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe33:3c4b/64 scope link 
       valid_lft forever preferred_lft forever
8: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fe:54:00:50:81:3f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe50:813f/64 scope link 
       valid_lft forever preferred_lft forever
9: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0a:d4:78:1f:cd:41 brd ff:ff:ff:ff:ff:ff
    inet 100.76.25.213/10 brd 100.127.255.255 scope global dynamic noprefixroute br0
       valid_lft 294sec preferred_lft 294sec
    inet6 2406:2d40:4100:8fb2:19c5:376e:1317:8ae1/64 scope global temporary dynamic 
       valid_lft 197sec preferred_lft 47sec
    inet6 2406:2d40:4100:8fb2:8d4:78ff:fe1f:cd41/64 scope global dynamic mngtmpaddr 
       valid_lft 197sec preferred_lft 47sec
    inet6 fe80::8d4:78ff:fe1f:cd41/64 scope link 
       valid_lft forever preferred_lft forever
10: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d2:da:46:a2:b2:3e brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.120/24 brd 192.168.1.255 scope global noprefixroute br1
       valid_lft forever preferred_lft forever
    inet6 fe80::d0da:46ff:fea2:b23e/64 scope link 
       valid_lft forever preferred_lft forever
ubuntu@ubuntu:/etc/netplan$ bridge link
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br1 state forwarding priority 32 cost 4 
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100 
ubuntu@ubuntu:/etc/netplan$ cat 01-network-manager-all.yaml
# Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager
  ethernets:
    enp2s0:
      dhcp4: true 
    enp3s0:
      dhcp4: true
  bridges:
    br0:
      dhcp4: true
      interfaces:
        - enp3s0
    br1:
      dhcp4: false
      addresses: [192.168.1.120/24]
      gateway4: 192.168.1.1
      nameservers:
        addresses: [8.8.8.8,8.8.8.4]
      interfaces:
        - enp2s0
ubuntu@ubuntu:/etc/netplan$ ip r
default via 100.64.0.1 dev br0 proto dhcp metric 425 
default via 192.168.1.1 dev br1 proto static metric 20426 
34.120.255.244 dev br0 proto dhcp scope link metric 425 
100.64.0.0/10 dev br0 proto kernel scope link src 100.76.25.213 metric 425 
169.254.0.0/16 dev virbr0 scope link metric 1000 linkdown 
192.168.1.0/24 dev br1 proto kernel scope link src 192.168.1.120 metric 426 
192.168.100.1 dev br0 proto dhcp scope link metric 425 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown
                                  V 1 Reply Last reply Reply Quote 0
                                  • V
                                    viragomann @eiger3970 0
                                    last edited by

                                    @eiger3970-0
                                    The "ip a" output doesn't look plausible to me. If a single NIC is slave in a bridge, I'd expect that the bridge takes over the MAC from the NIC.

                                    Also there should not be any IP settings on the NIC port. However, your enp2s0 shows IPv6 settings.

                                    Also I cannot see that any of your virtual interfaces is connected to one of the bridges. So it's expected that pinging doesn't work.
                                    Did you set the bridge name in VMM?

                                    E 1 Reply Last reply Reply Quote 1
                                    • E
                                      eiger3970 0 @viragomann
                                      last edited by

                                      @viragomann Maybe I'm missing the connection, but not sure what to do? I connect NIC0-br0 and NIC1-br1. Maybe I need something like NIC0-br0-vtnet0 and NIC1-br1-vtnet1?

                                      I create bridges for the VMM, in this order.
                                      1: Ubuntu 22.04 Host machine:
                                      sudo ip link set dev enp3s0 master br0
                                      sudo ip link set dev enp2s0 master br1

                                      2: Ubuntu 22.04 Host machine:
                                      /etc/netplan/01-network-manager-all.yaml

                                      # Let NetworkManager manage all devices on this system
network:
  version: 2
  renderer: NetworkManager
  ethernets:
    enp2s0:
      dhcp4: true 
    enp3s0:
      dhcp4: true
  bridges:
    br0:
      dhcp4: true
      interfaces:
        - enp3s0
    br1:
      dhcp4: false
      addresses: [192.168.1.120/24]
      gateway4: 192.168.1.1
      nameservers:
        addresses: [8.8.8.8,8.8.8.4]
      interfaces:
        - enp2s0

                                      3: Virtual Machine Manager 4.0.0:
                                      Virtual Network Interface > Network Source: Bridge device... > Device name: br0.
                                      Add Hardware > Virtual Network Interface > Network Source: Bridge device... > Device name: br1.

                                      V 1 Reply Last reply Reply Quote 0
                                      • V
                                        viragomann @eiger3970 0
                                        last edited by

                                        @eiger3970-0 said in What is the VMM Network selection for a LAN and WAN NIC:

                                        3: Virtual Machine Manager 4.0.0:
                                        Virtual Network Interface > Network Source: Bridge device... > Device name: br0.
                                        Add Hardware > Virtual Network Interface > Network Source: Bridge device... > Device name: br1.

                                        That's how I did it as well. But running "ip a", my virtual NICs show the bridges they're connected to:

                                        f5965430-28ed-4c59-a64a-7439ae25e5d5-image.png

                                        However, as already mentioned, I don't use the Network Manager.

                                        Do you use the host also for other purposes? If it's only for virtualization I'd consider to install Proxmox or alike on it.

                                        E 1 Reply Last reply Reply Quote 1
                                        • E
                                          eiger3970 0 @viragomann
                                          last edited by

                                          @viragomann I just use the host machine as a personal desktop.
                                          I previously ran Proxmox which ran nicely, however had to run Ubuntu as a vm.
                                          My hope is to run Ubuntu with better performance not being a vm any longer.

                                          E 1 Reply Last reply Reply Quote 0
                                          • E
                                            eiger3970 0 @eiger3970 0
                                            last edited by eiger3970 0

                                            @eiger3970-0 I've reinstalled the Ubuntu 22.04, VMM 4.0.0 and VM pfSense.
                                            Ubuntu 192.168.1.120 pings the Internet (very slow connection though), vm router and LAN devices.
                                            VM router 192.168.1.1 pings Ubuntu and LAN devices, however not the Internet.
                                            This tells me Internet is running via the Ubuntu 22.04 iptables, rather than via the vm router.
                                            Somehow I think I'm missing bridging the WAN interface NIC0-br0-vm router?
                                            Here's my latest configuration:

                                            root@ubuntu:/etc/netplan# cat 50-cloud-init.yaml
network:
  version: 2
  renderer: networkd

  ethernets:
    enp2s0:
      dhcp4: false
      dhcp6: false
    enp3s0:
      dhcp4: false
      dhcp6: false

  bridges:
    br0:
      interfaces: [enp3s0]
      dhcp4: true
      dhcp6: true
    br1:
      interfaces: [enp2s0]
      addresses: [192.168.1.120/24]
      # gateway4 is deprecated, use routes instead.
      routes:
      - to: default
        via: 192.168.1.1
        metric: 100
        on-link: true
      mtu: 1500
      nameservers:
        addresses: [8.8.8.8]
      parameters:
        stp: true
        forward-delay: 4
      dhcp4: no
      dhcp6: no        

                                            root@ubuntu:/etc/netplan# ip -c a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br1 state UP group default qlen 1000
    link/ether 1c:61:b4:6d:38:4f brd ff:ff:ff:ff:ff:ff
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether a8:a1:59:6e:1f:8b brd ff:ff:ff:ff:ff:ff
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:a9:81:30 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
8: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:d4:d4:20 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fed4:d420/64 scope link 
       valid_lft forever preferred_lft forever
9: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e2:73:dd:ac:ba:e0 brd ff:ff:ff:ff:ff:ff
    inet 100.76.25.213/10 metric 100 brd 100.127.255.255 scope global dynamic br0
       valid_lft 187sec preferred_lft 187sec
    inet6 2406:2d40:4100:8fb2:e073:ddff:feac:bae0/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 233sec preferred_lft 83sec
    inet6 fe80::e073:ddff:feac:bae0/64 scope link 
       valid_lft forever preferred_lft forever
10: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ce:08:2f:30:4a:71 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.120/24 brd 192.168.1.255 scope global br1
       valid_lft forever preferred_lft forever
    inet6 fe80::cc08:2fff:fe30:4a71/64 scope link 
       valid_lft forever preferred_lft forever
11: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:95:1e:e5 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe95:1ee5/64 scope link 
       valid_lft forever preferred_lft forever

                                            root@ubuntu:/etc/netplan# ip r
default via 192.168.1.1 dev br1 proto static metric 100 onlink 
default via 100.64.0.1 dev br0 proto dhcp src 100.76.25.213 metric 100 
1.1.1.1 via 100.64.0.1 dev br0 proto dhcp src 100.76.25.213 metric 100 
8.8.8.8 via 100.64.0.1 dev br0 proto dhcp src 100.76.25.213 metric 100 
34.120.255.244 dev br0 proto dhcp scope link src 100.76.25.213 metric 100 
100.64.0.0/10 dev br0 proto kernel scope link src 100.76.25.213 metric 100 
100.64.0.1 dev br0 proto dhcp scope link src 100.76.25.213 metric 100 
169.254.0.0/16 dev virbr0 scope link metric 1000 linkdown 
192.168.1.0/24 dev br1 proto kernel scope link src 192.168.1.120 
192.168.100.1 dev br0 proto dhcp scope link src 100.76.25.213 metric 100 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown 

                                            root@ubuntu:/etc/netplan# bridge link
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br1 state forwarding priority 32 cost 4 
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 4 
8: vnet3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100 
11: vnet4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br1 state forwarding priority 32 cost 100
                                            V 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.