Inbound NAT with Multi WAN broken with 21.02?
-
Hi,
I just upgraded my SG-3100 to 21.02. I have a setup with dual-wan connection which worked for years now.
Setup (simplyfied) is:
WAN1: DSL-PPOE (50 Mbit, fixed IPv4 address)
WAN2: Fiber-Connection with modem in full bridging mode between Internet an pfSense (900 Mbit, dynamic IPv4 address)
LAN: with mailserverThe default gateway is WAN2 because everything that not has to be reachable from outside should use the faster fiber connection.
I have inbound nat rules for accessing e.g. mailserver from the internet on WAN1.
After upgrade to 21.02 yesterday the Inbound NAT seemed to stop working. I can see incoming traffic over WAN1->pfSense->nat destination, but packets never travel back over WAN1 . They use the "default gateway" on WAN2. So no server is reachable from the internet anymore.
Is this a bug in 21.02 or do I have to add additional things/rules somewhere, which was not necessary with old release?
You help is very appreciated.
Greetz,
Michael -
Hello,
I have nearly the same problem after upgrading my XG-7100 1U to 21.02 :
The difference for me is that the answer from LAN server try to goes out to the correct interface for the correct WAN, but the outgoing answer is blocked by 1000000104 label "Default deny rule IPv4"...
The same exact same setup of course worked correctly in the previous version.I have tried some points of : http://doc.pfsense.org/index.php/Port_Forward_Troubleshooting with no success.
If i force a gateway on the interface rule, the block is not logged anylonger, but the flow still not work and it seems cant find it on tcpdump of interfaces, so it seems it's still blocked somewhere.Strangely, it works directly if i set the same Port Forward rule on an OpenVPN client interface (with direct public IP on it).
I did not find much differences on /tmp/rules.debug when i switch interface for the same rule (except the interface name of course ^^)Outbound NAT is pretty classic, manual rules for LAN to VPN Access, and automatic rules for WAN1/WAN2.
So im also interrested to know if there are extra rules needed by this version^^
Thanks in advance,
Adam -
Did you try deleting and recreating the inbound NAT rules?
I have found sometimes that pFSense gets confused over some of the rules if an interface changes, and solution is to delete and re-add. Maybe this update changed the way it reference interfaces which do not update some NAT rules correctly. -
Thanks for your answer.
Sadly, i tried to remove all port forwards, and recreate them and it still the same : outgoing answer to the interface of correct WAN blocked by the default deny out ...To summarize the flow :
WAN1>PfSense RDR:443>HTTPS_Server->pfSense-> blocked OUT default deny on WAN1.
(exactly same behavior on WAN2. but not on ovpnc....)
Edit : The WAN1's GW is the pfSense default GWI have also tried a completely new port forward, and its the same. Works only on ovpnc interface and not on WAN1/2.
I also tried switching to fully "Automatic outbound NAT" with no luck neither...
I dont understand why after upgrade to 21.02 it get "blocked" by the default rule, maybe the firewall dont recognize those answer packets as "RELATED,ESTABLISHED" or something like that ?
i kept verifying, the /tmp/rules.debug only move interface names when i change the a port forward rule, so no clues from here to me.
Note : my WAN1/2 interface addresses are private addresses to communicate with internet boxes that are configured as "DMZ (Nat1:1)" to the pfsense so the inbound public traffic is relayed as-is to pfsense. Obvously, the block private traffic is NOT CHECKED on theses interfaces^^, bogons block still "ON".
Note2 : ovpnc client that i can test have a direct public IP adress on it, that's the only difference with WAN1/2 i see right now.... (except from the interface names, lagg0.4090 vs ovpnc2)
Note3 : internet boxes configuration/FW was not changed, and the so called "DMZ"(nat1:1) feature work as intented.This pfsense masquerade outgoing LAN traffic correctly to boths WANs, VPNc, GW_GROUP so it seems not a "basic connectivity" issue
Thanks for your help,
Adam -
I am having the same problem except it's not an upgrade I just purchased an xg7100 running 21.02 and port forwarding is not working. I am using dual wan. Wan1 is a Starlink satellite so port forwarding won't work on it "CGNAT" so I have wan2 a 20MB WISP connection with static IP set up as a failover ( gateway settings Tier 1 Starlink tier 2 WISP) if I set them both to tier 1 port forwarding on WAN2 works or if I set the WISP to tier 1 and Starlink to Tier 2 then port forwarding will work. But I can't get port forwarding to work on WAN2 set to tier 2.
But my desire is to have Starlink be the primary and the wisp be backup and NAS access.
Any help would be much appreciated!
Nick
-
Big update since my precedent post : (I now run again 2.4.5p1 and evrything work fine)
1 : in fact, i got the exact same problem than the OP, the "Port Forwarding" works only for interface for the default GW interface (or higher priority on an GW_GROUP).
exactly like Nalasko and https://forum.netgate.com/topic/161157/multi-wan-port-forwarding-is-broke-in-pro-version ...2 : I was mistaken because i found another "bug" : OpenVPN client in pfsense getting by default the "push route defaut GW" from the VPN provider ; but the GUI general "routing" DID NOT SHOW IT !! (curl -o - http://checkip.dyndns.org to check it). Once checked the correct option in ovpnc ("dont pull routes" by memory) the GUI "routing" has correct default gw indication and modification.
Now i can confirm the working "port forward" follow the default GW in 21.02
(And i can confirm with the same config file the 2.4.5p1 work fine^^)3 : Finally I reinstalled (after asking the techsupport the installation image for my device : answer less than 2 minutes !!) the 2.4.5p1 image : after resinstallation, you will want to restore your backup config, but BEWARE : unplug all WANs/routed interface to the net because the automatic 'reinstall all packages' after a config upload/restore will try to install again the 21.02 release !! For my case, i tried to understand why i had several errors of missing libs to figure out that an update was ongoing without my intervention (php74, etc,etc).
What i done to prevent after a 3rd reinstall/restore : Restore config with only LAN/admin interface, wait long to loggin to pfSense (cause DNS not reachable, etc), switch update channel to "previous", cancel the automatic package resinstall, then reconnect WAN interfaces and eventually switch to stable (to see the "new 21.02" branch, and select back 2.4.5 DEPRECIATED until this NAT regression is fixed ^^@nalasko Ask the support for the 2.4.5p1 image for XG-7100 1U/DT throught the portal support as said in : https://docs.netgate.com/pfsense/en/latest/solutions/xg-7100/reinstall-pfsense.html : multiwan portforwarding work in this version
The "tiers" in pfSense has only sense (to my understanding^^) for outgoing/replies traffic.
Without the multi WAN bug we encounter, the portforward work on all WANs : its up to your client to define what is a primary and a backup (client side ? DNS roudrobin ? DNS update only? failover?)Adam
-
Same here.
-> deleted all port forwarding & nat rules (incoming and outgoing) and checked that all corresponding FW rules are also deleted -> reboot -> added new port forwarding rules with "add associated firewall rules" -> same behavior (not working if I'm not on the default gateway)
-> deleted again all port forwarding & nat rules and checked that all corresponding FW rules are deleted -> added new port forwarding rules without "add associated firewall rules" -> added the firewall rules manually -> same behavior (not working if I'm not on the default gateway)Even outbound nat is not working if I'm not on the default gateway.
Even more: system update shows "unable to check for updates" because my SG-3100 is searching for armv7 packages, which are definitely not existing on Netgate file server (checked this via browser an from ssh console of the SG-3100).
Too bad. For the meantime I've switched my default gateway to the slow DSL connection (with static IP) so I have the chance to get my eMail server up and running again. But now my 20x faster fiber connection is useless :-(
-
Glad to see I am not alone on this. I had heard some OpenVPN client issues could be present, so when that failed at first I thought I was going to have to go down that hole. Then I started getting down notices for my external services. That is when I realized none of my NAT rules were working properly.
Dual Wan config like the others. It looks like deleting and adding rules back is not working for anyone else so I guess we just sit and wait for a response?
For now I disabled CARP so that it would fail back to my older version unit.
-
Hmmm... I've seen that there is a hotfix (pfSense Plus 21.02-p1) for the nat problem... Installed it, but still the same error :-(
Someone else who already installed the hotfix?
Michael
-
Hello.
I have not read the hotfix release notes, but the ticket for this NAT regression is not assigned yet / still open : https://redmine.pfsense.org/issues/11436
So i doubt it is corrected in the hotfix...
(and the post https://forum.netgate.com/topic/161058/nat-issue-after-21-02-upgrade/4 seems to confirm that the hotfix still have the problem)For my setup, I will wait at least this ticket closed before trying again the 21.02 branch as the multi WAN inbound is mandatory for me.
Adam
-
I am also seeing this problem, on a SG-4860 running 21.02-p1.
Dual WAN, and NAT only works on the default WAN.Was hoping the hotfix would be the answer. Just installed it tonight, but it didn't fix the issue.
-Jeff
-
Has anyone contacted support on this? With enough people complaining in the forums I thought this would be an obvious and urgent issue to fix. I admit I haven't contacted support because my production environment required I quickly revert back to the old environment to keep things running. I figured with everything downgraded contacting support wouldn't be to useful other than to be another vote that this is a problem.
-
Hello,
It seems the community support was already contacted for this : ticket Regression 11436
It's priority is "very high" but the ticket has not yet been assigned since 17/02/2021.
And worse, the problem seems to begin with the 2.5 devel branch reported in december 2020 and not corrected since.@phatty i only have "community support" for my XG-7100 so can't contact the paid support for that problem. I've have reverted to 2.4.5p1 as well.
I can't imagine paid support customer not to use multi WAN inbound, so maybe they got an exclusive hotfix from paid support ?
Since this regression comes in 2.5 devel and was never corrected in devel branch, this should have been a no go for the pfSense+ 21.02 release on stable channel ...
Adam
-
Hi,
We also have the problem on a 2x XG-1537 cluster. This is very problematic.
Note that we have noticed a strange behavior with this bug. Simplified network topology :
FW2 (WAN IP) - Port fwd -> pfSense HA - Port fwd -> Srv.The FW2 is our second WAN. Port forwarding from this second WAN therefore no longer works because of the bug. While performing tests with OpenVPN and netcat (in UDP only), here is what we found:
- The first packet sent by the client reaches the server (port forwarding therefore works)
- The return packet sent by the server reaches the client. So far so good
- From there, all packets sent by the client arrive to the pfSense cluster (we see them with tcpdump on the link with the FW2) but they disappear silently and are never transmitted to the server
- On the other hand, all the packets sent from the server to the client arrive to the client (which makes sense since the bug concerns port forwarding)
Hope this bug is fixed quickly.
-
Hi,
I have same problem with SG-5100 and 21.02. Tried to install 2.4 but there is no download link on Netgate website. Why no comment from Netgate about this issue? Multiple WAN isn't uncommon these days.... so it affects lots of people.
Adam -
@nazgulix If you contact support they will provide you a link to download 2.4 firmware.
Also, assuming your NAT rules are focused on your primary WAN, if you set pfSense default routing to your primary WAN instead of auto things will work.
It is really disappointing such a bug was allowed to be released on a so called stable release. I also ran into an issue with OpenVPN not liking my CA Certificate causing me to downgrade security requirements to authenticate my users for the time being.
The band-aid of updating pfSense routing does work in my environment as I had hard coded NAT items to use my Fiber WAN vs cable modem.
-
@phatty Thanks for hint with default gateway. Unfortunately I have both "primary" WANs and there are diffrent NATs on each of them. I found pendrive with 2.4 but it didn't install ( some kernel errors ), don't have time for it now. Simply put SG-5100 on shelve and install fresh 2.4 on PC with 6 NICs. And will wait for a fix from Netgate.
Like you said that is release called "plus" and it supposed to be stable, but for now it's pfSense plus bugs... :(
-
Hello,
Just a little bit of patience, the ticket for this problem is in progress and is now assigned : Regression #11436
Seing the target version for the correction (2.5.1) it seems there will be no "hotfix" and will probably need to wait the release of the 2.5.1 (21.03 in pf+ ? maybe 21.04 ?^^)
Adam
-
Hi,
I must say im pretty disappointed from a "professional" firewall vendor...
The ticket for that "problem" is not really progressing (only ppl says +1 "i got the same problem in xxx/same situation").Until this regression not corrected, and given recently i have read This news (external phoronix link) i think i'll may wait pfSense to switch to FreeBSD 13.0(.1) before trying again the so called "stable" 21.02(21.XX because it's supposed to be versionned by date if im not wrong) branch on my XG-7100 firewall.
I suggest all that got the problem to do the same. (unless you have a "lab" and time to help the support of course ^^)
Adam -
I've been breaking my head over this..
Our site in france using a SG-3100 that also uses multi-wan has the same issue!