No connection to virtualized pfSense after add and reboot a 2nd WAN interface
-
@denbir Could it be an MTU of proxmox bridge vs MTU of net1 on pfsense issue?
-
@denbir I think the VLAN needs to also be configured (in proxmox) on the physical adapter that connects to the FTTH modem as a tagged VLAN, in order to create a trunk between the FTTH modem and pfsense because the path is:
FTTH ether -> Trunk for VLAN 22
Physical adapter on server running proxmox (VLAN aware?)
proxmox bridge for net1 (yes VLAN aware, as you said)
PfSense VirtIO iface (configure a VLAN with tag 22 on it)If the chain is broken in any adapter, all VLAN 22 packets are likely to be dropped.
-
@nightlyshark Pass-through is not possible because in the Server is only one hardware NIC with 4 integrated ports. If I understand this correct then can I use only dedicated Hardware for pass-through.
-
@nightlyshark I don't understand this behavior regarding the VLAN for 2nd WAN interface. I use in Proxmox with 3 interfaces.
Interface 1 for Management
Interface 2 for 1st Wan port and the new 2nd interface for 2nd Wan port with VLAN 22.
The 2nd Wan interface config should independent from the Management interface but this is not so. -
This post is deleted! -
@nightlyshark Sorry, this info was wrong. I've tested the config again and already after I add the new 2nd wan interface on the pfSense VM (with or without VLAN) after this step and and a reboot then is the pfSense not available.
-
@denbir Sorry for the delay. Did you enable the VLAN tag on the physical NIC that connects FTTH with proxmox?
-
@nightlyshark No worries. I am so happy that you support my challange :-).
The FTTH is direct connected with the physical port enp3s0 of the proxmox server. I've configured then a Linux Bridge vmbr2 and enabled VLAN aware.
I see no option to set direct on the physical port a VLAN tag.
I think this is not the problem because also if I add the new Network Device on the VM with or without a VLAN tag then is the pfSense after a reboot not reachable.
Why is here a dependency between the Management and the 2nd WAN? I don't understand this ...
Proxmox and other VMs on Proxmox are reachable only the pfSense is then not reachable.
-
@denbir I think you need to tag VLAN 22 on the enp3s0 iface for the traffic to trunk, not just vmbr2-net1. I think you can do it from CLI. (Unless you do not have local access to proxmox and are in danger of getting locked-out)
-
@denbir I think those bridges are not designed to handle ingress VLAN traffic, only inter-paravirtual and host VLANs.
-
@denbir I have a virtualized pfsense install on TrueNAS (bhyve hypervisor), and to have acceptable performance I had to add another NIC for TrueNAS itself and just passed the HP Broadcomm 4-port 1Gig-NIC directly to PfSense. Next step, slept like a baby.
-
@denbir said in No connection to virtualized pfSense after add and reboot a 2nd WAN interface:
@nightlyshark Pass-through is not possible because in the Server is only one hardware NIC with 4 integrated ports. If I understand this correct then can I use only dedicated Hardware for pass-through.
My i350-T4 cards show up with each individual interface as a separate entity in the Devices list under PCI Devices in Hardware. They show up as e.g. 0000:01:00.1, .2, .3 and .4 respectively.
Just make sure the box that reads "All Functions" is not ticked, otherwise the entire card will be allocated.Also, I believe that the port numbering seen by pfsense, like vtnet0, vtnet1 and so on, depends on which order you happen to add them in Proxmox. It's not the vmbr number that is showing up, it's net0, net1 etc. So if vmbr2 is added before vmbr0, vmbr2 will be vtnet0... At least that's what I think I'm seeing. I think that is what I am seeing in your picture earlier. LAN is vtnet1 which you gave a tag in Proxmox. With that setup, I think it you should put it on vmbr0. Perhaps you are mistaking the order of the ports on the card??
I would go in to the pfsense Console on Proxmox and make sure you have the three interfaces assigned as intended, and reassign if needed. One way forward could be to only assign the LAN and WAN as you had it set up before. Then the newly added interface will be possible to add from the UI before you finally save and reboot.
Wrt VLAN tags being set in Proxmox, that port will act the as an untagged port in a switch. Only VLAN 22 traffic will pass but pfsense will not be aware of the VLAN tag, since it's being stripped...
-
@gblenn Some specific NICs are registered as a single IOMMU domain and thus it is not possible to pass them on in hardware. My HPE Broadcom 4-Port Gigabit NIC had the same problem. It can either be passed through with all 4 ports or not at all. That is a hardware limitation and cannot be worked around.
-
@nightlyshark I was specific about my case being Intel (i350) since I wasn't sure about that. But I guess then that's another reason to be careful with Broadcom NICs.
That aside, I believe the issue is the following:
In this picture which @DenBir provided, I think the following applies:
WAN (vtnet0) is vmbr1
LAN (vtnet1) is vmbr2 which has VLAN 22 tag applied.
WANGF (pppoe0) is in fact vmbr0 (last one to be added in the list shown in Proxmox interface)
What might throw you off here @DenBir, is that things look ok in the list since you have an IP on the LAN port. But that will be there even if you don't actually connect a cable to that port (it is set by pfsense). Proxmox on the other hand, will not pass any other traffic than VLAN 22 on that port, which is why you lose access to pfsense.
If you change it so that tag 22 is on vmbr0 instead, I believe you should get it working as intended.
BTW, Proxmox lists the Bridge ports in order - vmbr0, 1, 2 etc. The physical ports used for this list are picked top to bottom on my PCIe cards. Meaning vmbr0 is the top port on my multi port cards (4 or 2 port).
-
Thank you very much for all of your feedback! I appreciate that.
I've tried this config (vmbr0 = VLAN tag 22) but the same behavior :-(
I've made also this table to better understand the environment:
-
@denbir Try removing all tags and setting a PVID of 1 on proxmox while keeping VLAN-Aware activated and in PfSense create a VLAN adapter (22) on the WAN virtual port (not on the PPPoE iface) and create a PPPoE on the VLAN adapter.
-
@denbir said in No connection to virtualized pfSense after add and reboot a 2nd WAN interface:
Thank you very much for all of your feedback! I appreciate that.
I've tried this config (vmbr0 = VLAN tag 22) but the same behavior :-(
I've made also this table to better understand the environment:
Did you stop the VM and then start it again after changing the tag assignment??
Adding a port from Proxmox should not really mess things up for you in this way...
If you look in the network list that you find under pve (not the pfsense VM). What does that show? Are those the only four physical ports you have, is there no Motherboard port?Regarding your table, I'm thinking that you need to revise it slightly...
The Bridge Port numbering and Proxmox physical port numbers would normally match. So vmbr1 and vmbr2 will be enpNs1 and enpNs2 respectively. N will depend on the cards you have, can be 1, 10 or any number basically. But it should be the same for all ports on one single card. At least that is what I am seeing in my installations.
The numbering you have for net0, net1 and so on, is entirely based on the order by which you add them to the VM. This is also the same numbering that pfsense will see, although naming is based on the driver used, in this case virtio gives: net0=vtnet0, net1=vtnet1 and so on.
Based on this, I think the list would be:
Bridge port - Physical port - Description - VMinterface - pfsense port
vmbr1 ------ enp1s1 --------FW WAN ---- net0 ---------- vtnet0
vmbr2 ------ enp1s2 -------- LAN --------- net1 ---------- vtnet1
vmbr0 ------ enp1s0 --------WAN-GF ----- net2 ---------- vtnet2Essentially you are using all but the bottom port on the card?
Perhaps you should consider redoing the port assignments in Proxmox from scratch. And in doing so making sure you connect them physically in the same order, starting with WAN in the top port in the card (vtnet0). LAN in second and the new WAN - VLAN 22 in third. And in Proxmox you make sure to add them to the VM in that same order, vmbr0, vmbr1 and finally you add vmbr2 with VLAN 22 tag.
This will change the list of course but then you have the numbering in order. When you boot up pfsense, open the console on Proxmox to check the assignments there as well... You should still see them in the same order there, vtnet0=WAN, vtnet1=LAN and vtnet2=WAN-GF
-
@denbir On more thing... perhaps I'm misunderstanding a bit how your ISP has things set up, but do you really need to worry about VLAN at all?? I get it if you have IPTV from them, that it would be on a separate VLAN and you need to forward that on the internal side. But for the connection to the ISP, do you even need to care? A port on Proxmox will trunk all VLAN tags if you make it VLAN aware.
-
Short update.
I've decided to reinstall pfSense from scratch without proxmox virtualization. I've configured again VLAN 22 in pfSense and now the new WAN connection runs fine (also after a reboot).Thanks everyone to investigate my issue and the challenge.
-
