Struggling with Multi-WAN on incoming traffic - Please help

[Ascar]

Hello, everyone. Please help me with the Multi-WAN configuration. Can't figure it out myself.

I run pfSense 2.7.2 in a VM on top of a server collocated in a professional datacenter. The service provider has 3 different public subnets from which I got 3 different IP addresses (addresses are modified/made up for the purpose of obfuscation) - 11.22.33.254, 11.22.34.254 and 11.22.35.254. The pfSense VM has 4 virtual NICs. The first 3 vNICs are assigned these public IP addresses and the first vNIC is defined as WAN, so it is the default gateway. The other 2 IP Address / vNIC pares are also set up as gateways, so they are essentially WAN2 and WAN3. The last vNICs is assigned the role of LAN interface with IP address 192.168.20.254.

Traffic flows perfectly in and out of WAN1 (default gateway). Policy based routing works fine also, for the sake of experiment and testing I made some firewall rules to push traffic from a specific host or to a specific destination through any of the available gateways and PBR works.

The problem I have and that I can't crack myself is routing of incoming traffic destined at either WAN2 or WAN3. Again, on the purpose of checking and testing I allowed ICMP Echo on both interfaces and I can ping them. However, when I set up port forwarding on WAN2 or WAN3 to forward any port (e.g. TCP22) to some host on the LAN (associated firewall rules created and enabled) the traffic does not get through and packets are dropped. I see in the logs that packets hit the WAN2 interface but they are all dropped by the default deny rule IPv4 1000000103 with TCP:S flag. I have tried creating firewall rules manually, NAT associated rules, all kinds of settings and parameters, disabling firewall from the console just for the sake of checking whether connection would establish when the filter is disabled. The default deny rule takes precedence...

The settings I tried:
Advanced -> Firewall & NAT -> Firewall State Policy
Advanced -> Firewall & NAT -> Static Route Filtering -> Bypass firewall rules for traffic on the same interface
Advanced -> Firewall & NAT -> Disable Negate rules
Advanced -> Miscellaneous -> Load Balancing -> Use Sticky Connections
I also tried floating firewall rules

What else I have not done? Can I achieve in general what I am trying to do?

For me this use scenario is important so I went an extra mile or two. Sharing my discoveries.

Trying to reproduce the issue I made a new VM and installed OPNsense into. Prior to that I disabled the WAN2 and WAN3 interfaces on the pfSense VM and thus released the IP addresses. Then I set up additional WAN interfaces accordingly on OPNsense. 2 simple NAT forward rules to forward port 22 to the same VM on the LAN. Surprise, surprise - everything worked without a hitch and firewall configuration voodoo on both IP addresses.

My next guess was that - well, I have a botched configuration on pfSense. Made a new VM and fresh install of pfSense 2.7.2, so it is a pristine new firewall without anything configured. Added additional WAN2 and WAN3 interfaces. Added simple port forwarding rules and nothing works just like on the production firewall. So, it is not the misconfigured firewall, something is inside the design of pfSense.

PBR, policy-based routing works without problem, outward traffic may be pushed through any gateway on pfSense. When I need to route inward traffic through a WAN interface other than default gateway nothing works.

Thanks very much in advance

[viragomann]

@Ascar
Are the additional subnets routed to the primary IP by any chance?

I see in the logs that packets hit the WAN2 interface but they are all dropped by the default deny rule IPv4 1000000103 with TCP:S flag.

Sure that it's the correct interface in the log?

[Ascar]

@viragomann Yes, correct interface, WAN2, when trying to connect.

[viragomann]

@Ascar
Then the rule might be wrong anyhow, so that it doesn't match.