pfSense on Proxmox with questionable connectivity (Port Forwarding)

viragomann

@whytf said in pfSense on Proxmox with questionable connectivity (Port Forwarding):

Port forwarding is setup the usual way for load balanced WANs, interface Gateway Group, source any

Which one interface group or gateway group?
To be honest, I don't know any option to state a gateway group in port forwarding, and interface group would not be the correct way to do it.

whytf

@viragomann I have been running gateway group as NAT interface for some time now without any issues, I made sure to retest my setup using NAT interface as either one ISP or other but still with same issue.

darcey

@whytf I am not suggesting it as a fix, but what happens if you change destination to 'any' in the forward rule?

viragomann

@whytf
As mentioned, port forwarding on an interface group is the wrong way to do it.

However, it's not on the NAT rule, in fact it's the firewall rule, which has to be defined on each unique interface separately.

This means, you could also leave the NAT rule on the group, but have disable the filter rule association and create the rule manually on each interface.
Or just define the port forwardings with associated filter rules on each interface separately.

At all events, there must no pass rule on an interface group or a floating pass rule match to the forwarded traffic!

whytf

@darcey said

@whytf I am not suggesting it as a fix, but what happens if you change destination to 'any' in the forward rule?

So I tried setting it to any when using interface group and separately but nothing changed, nothing unusual in system logs as well.

@viragomann said

@whytf
As mentioned, port forwarding on an interface group is the wrong way to do it.

However, it's not on the NAT rule, in fact it's the firewall rule, which has to be defined on each unique interface separately.

This means, you could also leave the NAT rule on the group, but have disable the filter rule association and create the rule manually on each interface.
Or just define the port forwardings with associated filter rules on each interface separately.

At all events, there must no pass rule on an interface group or a floating pass rule match to the forwarded traffic!

It's really interesting that this solved port forwarding for pppoe ISP1 but the dhcp ISP2 port forwarding still doesn't work at all, some time ago I have confirmed with ISP2 on their part that they use 1:1 NAT, however I can't think of anything that could affect this just by using linux bridges, which is just a switch. Hmm any ideas?

viragomann

@whytf said in pfSense on Proxmox with questionable connectivity (Port Forwarding):

but the dhcp ISP2 port forwarding still doesn't work at all,

And you say, it worked if you pass through the NIC?
So why don't you do it? Seems you have enough.

I don't think, that it has something to do with the virtual bridges, at least as long as Proxmox has no IP in these subnets.

whytf

@viragomann said in pfSense on Proxmox with questionable connectivity (Port Forwarding):

And you say, it worked if you pass through the NIC?
So why don't you do it? Seems you have enough.

Yes the passthrough works, however I am quite limited by CPU and RAM, which could be solved by using linux drivers, average latency is around 50% lower too, only problem is with that one dhcp ISP

Also I intended to have higher throughput for my VMs running on the "router", if I remember correctly I hit around 21Gb/s when using virtIO between VMs, I was planning to create a bond between the "router" and NAS, both are connected to a switch with enough free ports, but first I need to get it to work properly before experimenting with bonds and such.

I don't think, that it has something to do with the virtual bridges, at least as long as Proxmox has no IP in these subnets.

Yep, proxmox has IP in LAN subnet, only difference between passing through and having virtual NIC is just kernel, driver and one extra hop as it is in virtual switch, worked with pppoe without issues so I doubt it could be kernel or driver and extra hop isn't that bad to cause something like this.

viragomann

@whytf
Did you disable the hardware checksum offloading?

whytf

@viragomann said in pfSense on Proxmox with questionable connectivity (Port Forwarding):

@whytf
Did you disable the hardware checksum offloading?

Yes I have disabled it in pfsense

As for proxmox ethtool -K enp1s0f0 rx off tx off, ethtool -K vmbr1 rx off tx off

viragomann

@whytf
All right.

whytf

Bump

whytf

For future reference I made a summarized the all the information possibly required:

My setup:

Proxmox version: 8.1.4
pfSense version: 2.7.2
NIC: i340-t4, i219 (motherboard)
Network configuration:
vmbr0 is assigned to LAN in pfsense and all other VMs in proxmox, it also has slaved physical port (i340-t4) that connects to rest of the lan
vmbr1 is assigned to WAN in pfsense and it has slaved physical port (i340-t4) to ISP1(DHCP)
vmbr2 is assigned to WAN2 in pfsense and it has slaved physical port (i340-t4) to ISP2(PPPoE)
vmbr4 is assigned for proxmox management/cluster only and it has slaved physical port (i219) that connects to same physical switch as vmbr0/rest of the lan

The issue:

Port forwarding works when using NIC passthrough, but not when using virtIO
Specifically, port forwarding doesn't work for the DHCP ISP connection when using virtIO, but does work with PPPoE ISP2

I have tried:

disable hardware offloading in pfsense
ethtool -K XXXX rx off tx off for physical ports as well as vmbr(0-4) on proxmox
manually changing MAC Addresses on vmbr(0-4) in case there would be a conflict, especially vmbr1 having same MAC as the physical interface
This is my /etc/network/interfaces with manual MAC Addresses, to test without that I just comment out the vmbr hwaddress lines:

auto lo
iface lo inet loopback

auto enp1s0f2
iface enp1s0f2 inet manual

iface enp1s0f3 inet manual

iface enp1s0f0 inet manual
        hwaddress XXXXXXXXXX

iface enp1s0f1 inet manual

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet manual
        bridge-ports enp1s0f3
        bridge-stp off
        bridge-fd 0
        hwaddress 90:e2:ba:37:0d:a0
#LAN

auto vmbr1
iface vmbr1 inet manual
        bridge-ports enp1s0f0
        bridge-stp off
        bridge-fd 0
        hwaddress 90:e2:ba:37:0d:a1
#Antik

auto vmbr2
iface vmbr2 inet manual
        bridge-ports enp1s0f1
        bridge-stp off
        bridge-fd 0
        hwaddress 90:e2:ba:37:0d:a2
#Telekom

auto vmbr4
iface vmbr4 inet static
        address 192.168.0.70/16
        gateway 192.168.0.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        hwaddress 50:65:f3:48:34:a4
#PVE