No connection to virtualized pfSense after add and reboot a 2nd WAN interface
-
@denbir Configuring the VLAN tag 22 on the hypervisor, means only packets with a VLAN tag of 22 pass to the net1->virtio from the hypervisor vswitch. Just in case you ever need to configure it like this.
-
@nightlyshark OK but VLAN 22 is necessary from the ISP. That is why is configured on the Hypervisor. I'll try this on the pfSense direct to configure VLAN 22 for the 2nd WAN port.
-
@denbir Just had a thought, did you set vmbr2 as VLAN-aware?
-
-
@nightlyshark Yes, VLAN aware was already configured.
-
@denbir I don't know enough about proxmox to be of help there, but there must be a way to untag the VLAN 22 packets before they reach pfsense, which would only do PPPoE without being aware of a VLAN.
Or just pass-through an adapter as hardware to pfsense and connect from there? In case your hardware allows you to, of course. This would also greatly improve pfsense performance, as it uses hardware capabilities of network adapters heavily.
-
@denbir Could it be an MTU of proxmox bridge vs MTU of net1 on pfsense issue?
-
@denbir I think the VLAN needs to also be configured (in proxmox) on the physical adapter that connects to the FTTH modem as a tagged VLAN, in order to create a trunk between the FTTH modem and pfsense because the path is:
FTTH ether -> Trunk for VLAN 22
Physical adapter on server running proxmox (VLAN aware?)
proxmox bridge for net1 (yes VLAN aware, as you said)
PfSense VirtIO iface (configure a VLAN with tag 22 on it)If the chain is broken in any adapter, all VLAN 22 packets are likely to be dropped.
-
@nightlyshark Pass-through is not possible because in the Server is only one hardware NIC with 4 integrated ports. If I understand this correct then can I use only dedicated Hardware for pass-through.
-
@nightlyshark I don't understand this behavior regarding the VLAN for 2nd WAN interface. I use in Proxmox with 3 interfaces.
Interface 1 for Management
Interface 2 for 1st Wan port and the new 2nd interface for 2nd Wan port with VLAN 22.
The 2nd Wan interface config should independent from the Management interface but this is not so. -
This post is deleted! -
@nightlyshark Sorry, this info was wrong. I've tested the config again and already after I add the new 2nd wan interface on the pfSense VM (with or without VLAN) after this step and and a reboot then is the pfSense not available.
-
@denbir Sorry for the delay. Did you enable the VLAN tag on the physical NIC that connects FTTH with proxmox?
-
@nightlyshark No worries. I am so happy that you support my challange :-).
The FTTH is direct connected with the physical port enp3s0 of the proxmox server. I've configured then a Linux Bridge vmbr2 and enabled VLAN aware.
I see no option to set direct on the physical port a VLAN tag.
I think this is not the problem because also if I add the new Network Device on the VM with or without a VLAN tag then is the pfSense after a reboot not reachable.
Why is here a dependency between the Management and the 2nd WAN? I don't understand this ...
Proxmox and other VMs on Proxmox are reachable only the pfSense is then not reachable.
-
@denbir I think you need to tag VLAN 22 on the enp3s0 iface for the traffic to trunk, not just vmbr2-net1. I think you can do it from CLI. (Unless you do not have local access to proxmox and are in danger of getting locked-out)
-
@denbir I think those bridges are not designed to handle ingress VLAN traffic, only inter-paravirtual and host VLANs.
-
@denbir I have a virtualized pfsense install on TrueNAS (bhyve hypervisor), and to have acceptable performance I had to add another NIC for TrueNAS itself and just passed the HP Broadcomm 4-port 1Gig-NIC directly to PfSense. Next step, slept like a baby.
-
@denbir said in No connection to virtualized pfSense after add and reboot a 2nd WAN interface:
@nightlyshark Pass-through is not possible because in the Server is only one hardware NIC with 4 integrated ports. If I understand this correct then can I use only dedicated Hardware for pass-through.
My i350-T4 cards show up with each individual interface as a separate entity in the Devices list under PCI Devices in Hardware. They show up as e.g. 0000:01:00.1, .2, .3 and .4 respectively.
Just make sure the box that reads "All Functions" is not ticked, otherwise the entire card will be allocated.Also, I believe that the port numbering seen by pfsense, like vtnet0, vtnet1 and so on, depends on which order you happen to add them in Proxmox. It's not the vmbr number that is showing up, it's net0, net1 etc. So if vmbr2 is added before vmbr0, vmbr2 will be vtnet0... At least that's what I think I'm seeing. I think that is what I am seeing in your picture earlier. LAN is vtnet1 which you gave a tag in Proxmox. With that setup, I think it you should put it on vmbr0. Perhaps you are mistaking the order of the ports on the card??
I would go in to the pfsense Console on Proxmox and make sure you have the three interfaces assigned as intended, and reassign if needed. One way forward could be to only assign the LAN and WAN as you had it set up before. Then the newly added interface will be possible to add from the UI before you finally save and reboot.
Wrt VLAN tags being set in Proxmox, that port will act the as an untagged port in a switch. Only VLAN 22 traffic will pass but pfsense will not be aware of the VLAN tag, since it's being stripped...
-
@gblenn Some specific NICs are registered as a single IOMMU domain and thus it is not possible to pass them on in hardware. My HPE Broadcom 4-Port Gigabit NIC had the same problem. It can either be passed through with all 4 ports or not at all. That is a hardware limitation and cannot be worked around.
-
@nightlyshark I was specific about my case being Intel (i350) since I wasn't sure about that. But I guess then that's another reason to be careful with Broadcom NICs.
That aside, I believe the issue is the following:
In this picture which @DenBir provided, I think the following applies:
WAN (vtnet0) is vmbr1
LAN (vtnet1) is vmbr2 which has VLAN 22 tag applied.
WANGF (pppoe0) is in fact vmbr0 (last one to be added in the list shown in Proxmox interface)
What might throw you off here @DenBir, is that things look ok in the list since you have an IP on the LAN port. But that will be there even if you don't actually connect a cable to that port (it is set by pfsense). Proxmox on the other hand, will not pass any other traffic than VLAN 22 on that port, which is why you lose access to pfsense.
If you change it so that tag 22 is on vmbr0 instead, I believe you should get it working as intended.
BTW, Proxmox lists the Bridge ports in order - vmbr0, 1, 2 etc. The physical ports used for this list are picked top to bottom on my PCIe cards. Meaning vmbr0 is the top port on my multi port cards (4 or 2 port).
