pfSense on Proxmox with questionable connectivity (Port Forwarding)
-
@viragomann I have been running gateway group as NAT interface for some time now without any issues, I made sure to retest my setup using NAT interface as either one ISP or other but still with same issue.
-
@whytf I am not suggesting it as a fix, but what happens if you change destination to 'any' in the forward rule?
-
@whytf
As mentioned, port forwarding on an interface group is the wrong way to do it.However, it's not on the NAT rule, in fact it's the firewall rule, which has to be defined on each unique interface separately.
This means, you could also leave the NAT rule on the group, but have disable the filter rule association and create the rule manually on each interface.
Or just define the port forwardings with associated filter rules on each interface separately.At all events, there must no pass rule on an interface group or a floating pass rule match to the forwarded traffic!
-
@darcey said
@whytf I am not suggesting it as a fix, but what happens if you change destination to 'any' in the forward rule?
So I tried setting it to any when using interface group and separately but nothing changed, nothing unusual in system logs as well.
@viragomann said
@whytf
As mentioned, port forwarding on an interface group is the wrong way to do it.However, it's not on the NAT rule, in fact it's the firewall rule, which has to be defined on each unique interface separately.
This means, you could also leave the NAT rule on the group, but have disable the filter rule association and create the rule manually on each interface.
Or just define the port forwardings with associated filter rules on each interface separately.At all events, there must no pass rule on an interface group or a floating pass rule match to the forwarded traffic!
It's really interesting that this solved port forwarding for pppoe ISP1 but the dhcp ISP2 port forwarding still doesn't work at all, some time ago I have confirmed with ISP2 on their part that they use 1:1 NAT, however I can't think of anything that could affect this just by using linux bridges, which is just a switch. Hmm any ideas?
-
@whytf said in pfSense on Proxmox with questionable connectivity (Port Forwarding):
but the dhcp ISP2 port forwarding still doesn't work at all,
And you say, it worked if you pass through the NIC?
So why don't you do it? Seems you have enough.I don't think, that it has something to do with the virtual bridges, at least as long as Proxmox has no IP in these subnets.
-
@viragomann said in pfSense on Proxmox with questionable connectivity (Port Forwarding):
And you say, it worked if you pass through the NIC?
So why don't you do it? Seems you have enough.Yes the passthrough works, however I am quite limited by CPU and RAM, which could be solved by using linux drivers, average latency is around 50% lower too, only problem is with that one dhcp ISP
Also I intended to have higher throughput for my VMs running on the "router", if I remember correctly I hit around 21Gb/s when using virtIO between VMs, I was planning to create a bond between the "router" and NAS, both are connected to a switch with enough free ports, but first I need to get it to work properly before experimenting with bonds and such.
I don't think, that it has something to do with the virtual bridges, at least as long as Proxmox has no IP in these subnets.
Yep, proxmox has IP in LAN subnet, only difference between passing through and having virtual NIC is just kernel, driver and one extra hop as it is in virtual switch, worked with pppoe without issues so I doubt it could be kernel or driver and extra hop isn't that bad to cause something like this.
-
@whytf
Did you disable the hardware checksum offloading? -
@viragomann said in pfSense on Proxmox with questionable connectivity (Port Forwarding):
@whytf
Did you disable the hardware checksum offloading?Yes I have disabled it in pfsense
As for proxmox
ethtool -K enp1s0f0 rx off tx off,ethtool -K vmbr1 rx off tx off -
@whytf
All right. -
Bump
-
For future reference I made a summarized the all the information possibly required:
My setup:
- Proxmox version: 8.1.4
- pfSense version: 2.7.2
- NIC: i340-t4, i219 (motherboard)
- Network configuration:
- vmbr0 is assigned to LAN in pfsense and all other VMs in proxmox, it also has slaved physical port (i340-t4) that connects to rest of the lan
- vmbr1 is assigned to WAN in pfsense and it has slaved physical port (i340-t4) to ISP1(DHCP)
- vmbr2 is assigned to WAN2 in pfsense and it has slaved physical port (i340-t4) to ISP2(PPPoE)
- vmbr4 is assigned for proxmox management/cluster only and it has slaved physical port (i219) that connects to same physical switch as vmbr0/rest of the lan
The issue:
- Port forwarding works when using NIC passthrough, but not when using virtIO
- Specifically, port forwarding doesn't work for the DHCP ISP connection when using virtIO, but does work with PPPoE ISP2
I have tried:
- disable hardware offloading in pfsense
- ethtool -K XXXX rx off tx off for physical ports as well as vmbr(0-4) on proxmox
- manually changing MAC Addresses on vmbr(0-4) in case there would be a conflict, especially vmbr1 having same MAC as the physical interface
- This is my /etc/network/interfaces with manual MAC Addresses, to test without that I just comment out the vmbr hwaddress lines:
auto lo iface lo inet loopback auto enp1s0f2 iface enp1s0f2 inet manual iface enp1s0f3 inet manual iface enp1s0f0 inet manual hwaddress XXXXXXXXXX iface enp1s0f1 inet manual iface eno1 inet manual auto vmbr0 iface vmbr0 inet manual bridge-ports enp1s0f3 bridge-stp off bridge-fd 0 hwaddress 90:e2:ba:37:0d:a0 #LAN auto vmbr1 iface vmbr1 inet manual bridge-ports enp1s0f0 bridge-stp off bridge-fd 0 hwaddress 90:e2:ba:37:0d:a1 #Antik auto vmbr2 iface vmbr2 inet manual bridge-ports enp1s0f1 bridge-stp off bridge-fd 0 hwaddress 90:e2:ba:37:0d:a2 #Telekom auto vmbr4 iface vmbr4 inet static address 192.168.0.70/16 gateway 192.168.0.1 bridge-ports eno1 bridge-stp off bridge-fd 0 hwaddress 50:65:f3:48:34:a4 #PVE