No internet after upgrading to 2.7.1-ce from 2.7.0-ce

[coldfire7]

No internet after upgrading to 2.7.1-ce from 2.7.0-ce. 3.5 hours were wasted on the upgrade (super slow 100-200kbps download speed from the pkg server).
Can ping and access the firewall but cannot ping anything outbound to the internet. The firewall seems to have internet connectivity and can ping/traceroute from the firewall to the internet.

[ramikilany]

Same problem happened with me now. I just upgraded to 2.7.1-ce and restarted the server 3 times same result. I can access the firewall through real IP and from the firewall I can ping and trace rout also can download new packages. But all my internal network has no internet access.

Switched to the new DHCP Server Backend Kea still have the same problem.

I hope to find the solution asap, my hospital has no internet.

[SteveITS]

@ramikilany check for a default route but if pfSense can get out that’s likely not it.

Check outbound NAT, try automatic if it isn’t set to that.

[ramikilany]

The route exist and configured.

The NAT outbound is already set to automatic (please check the image)

The weird thing I did not do anything in my configuration just upgrade the firewall from 2.7.0 to 2.7.1.

Please any new help?

[coldfire7]

@SteveITS I don't think it's NAT-related, that was the first thing I checked after the upgrade.

[chris1284]

@coldfire7 Same here, after a time working no network for clients that use dhcp. Services are running and I can do anything over vpn. Kea is not active and I will go back to 2.7.0.

[SteveITS]

@chris1284 re: DHCP, see if it’s this: https://redmine.pfsense.org/issues/15011

[stephenw10]

@ramikilany said in No internet after upgrading to 2.7.1-ce from 2.7.0-ce:

The NAT outbound is already set to automatic (please check the image)

Do you actually see the expected auto OBT rules created though?

When you try to ping from a client device how does it fail?

What states are created on that firewall for that ping?

[coldfire7]

@stephenw10 In my case NAT rules are manually added, I also tried automatic and it showed a bunch of automatically created rules at the bottom. I can ping/trace the firewall from LAN and can ping/trace anything outbound from the firewall but cannot ping/trace anything outbound from LAN. Also, the GUI kinda felt sluggish and was taking much longer to load compared to 2.7.0.

[SteveITS]

@coldfire7 said in No internet after upgrading to 2.7.1-ce from 2.7.0-ce:

can ping/trace anything outbound from the firewall

From Diagnostics/Traceroute or /Ping if you select the Source Address of LAN does it succeed?

An open state would indicate if the firewall is allowing the outbound connection.

[stephenw10]

@coldfire7 said in No internet after upgrading to 2.7.1-ce from 2.7.0-ce:

cannot ping/trace anything outbound from LAN.

How does it fail when you try?

What states are created in the firewall when you do?

[coldfire7]

@SteveITS

[coldfire7]

@stephenw10

[stephenw10]

Ok, so there's no outbound NAT happening on the DOT interface.

Is there an auto rule being created for that? Or a manual rule in place?

[coldfire7]

@stephenw10 Manual and auto both didn't work. Since you said no outbound NAT is working, I started testing by turning things on and off to see if I could get it back working again and finally, I have found the issue. The firewall has 2 WAN interfaces. I disabled number 2 a few days ago since the connection was offline due to a fibre cut but the NAT rules for that interface are still present, those rules are causing the NAT to stop working when the interface is disabled. If I disable those rules or re-enable the WAN interface NAT starts working again. This bug/issue started from CE v2.7.1. In CE v2.7.0 and previous versions, it was working fine.

[ramikilany]

Same things for me, I worked also with a network engineer and test a lot of things in our network up to the firewall. The internet connection is blocked in the LAN network, so we switched to other vlan from the core switch, it works for 5 minutes. after 5 minutes it block the internet connection try other vlan works and stopped working after a time. After more than 20 hours we have problem in the DHCP network the problem was only in the internet then it continue to not connect to the servers and between each others.

I downgraded to 2.7.0 and of course other issues happens:
1- pfblocker is not installed and cannot installed
2- the package manager stops showing the software to install (try to reinstall pfblocker)
3- the image disappear (not huge problem but just to mention)
4- i have crash in the system:

Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 RELENG_2_7_0-n255866-686c8d3c1f0: Wed Jun 28 04:21:19 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/obj/amd64/LwYAddCr/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/sources/FreeBSD-src-REL

Crash report details:

PHP Errors:
[21-Nov-2023 10:40:42 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'ftp.so' (tried: /usr/local/lib/php/20220829/ftp.so (Shared object "libssl.so.30" not found, required by "ftp.so"), /usr/local/lib/php/20220829/ftp.so.so (Cannot open "/usr/local/lib/php/20220829/ftp.so.so")) in Unknown on line 0
[21-Nov-2023 10:45:42 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'ftp.so' (tried: /usr/local/lib/php/20220829/ftp.so (Shared object "libssl.so.30" not found, required by "ftp.so"), /usr/local/lib/php/20220829/ftp.so.so (Cannot open "/usr/local/lib/php/20220829/ftp.so.so")) in Unknown on line 0
[21-Nov-2023 10:50:43 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'ftp.so' (tried: /usr/local/lib/php/20220829/ftp.so (Shared object "libssl.so.30" not found, required by "ftp.so"), /usr/local/lib/php/20220829/ftp.so.so (Cannot open "/usr/local/lib/php/20220829/ftp.so.so")) in Unknown on line 0
[21-Nov-2023 10:55:43 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'ftp.so' (tried: /usr/local/lib/php/20220829/ftp.so (Shared object "libssl.so.30" not found, required by "ftp.so"), /usr/local/lib/php/20220829/ftp.so.so (Cannot open "/usr/local/lib/php/20220829/ftp.so.so")) in Unknown on line 0
[21-Nov-2023 11:00:43 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'ftp.so' (tried: /usr/local/lib/php/20220829/ftp.so (Shared object "libssl.so.30" not found, required by "ftp.so"), /usr/local/lib/php/20220829/ftp.so.so (Cannot open "/usr/local/lib/php/20220829/ftp.so.so")) in Unknown on line 0
[21-Nov-2023 11:05:44 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'ftp.so' (tried: /usr/local/lib/php/20220829/ftp.so (Shared object "libssl.so.30" not found, required by "ftp.so"), /usr/local/lib/php/20220829/ftp.so.so (Cannot open "/usr/local/lib/php/20220829/ftp.so.so")) in Unknown on line 0

No FreeBSD crash data found

[SteveITS]

@ramikilany when you installed 2.7.0 did you change your update branch to Previous before trying to install any packages (from the later version)?

[stephenw10]

@coldfire7 said in No internet after upgrading to 2.7.1-ce from 2.7.0-ce:

I disabled number 2 a few days ago since the connection was offline due to a fibre cut but the NAT rules for that interface are still present, those rules are causing the NAT to stop working when the interface is disabled.

NAT rules can be present, they don't direct traffic. NAT rules only translate traffic that is already leaving that interface.

So you can see in the states opened by that ping the traffic leaving on the DOT interface but not being NAT'd. There are probably no NAT rules on DOT. Or at least none that match. NAT rule on the DHK interface have no effect there. What NAT rules do you have on DOT?

[chris1284]

@SteveITS said in No internet after upgrading to 2.7.1-ce from 2.7.0-ce:

@chris1284 re: DHCP, see if it’s this: https://redmine.pfsense.org/issues/15011

don't know, was an unacceptable situation so reinstall and config restore was a quick solution

[coldfire7]

@stephenw10 NAT rules are present for all the WAN and VPN interfaces.

I checked this like 10 times and I'm 100% sure. If I disable the WAN 2 (DHK) interface while the WAN 2 NAT rules are present, NAT stops working for all interfaces. I either have to re-enable the WAN 2 interface or remove/disable the WAN 2 NAT rules to get it back working again.