No internet after upgrading to 2.7.1-ce from 2.7.0-ce

coldfire7

@stephenw10 Manual and auto both didn't work. Since you said no outbound NAT is working, I started testing by turning things on and off to see if I could get it back working again and finally, I have found the issue. The firewall has 2 WAN interfaces. I disabled number 2 a few days ago since the connection was offline due to a fibre cut but the NAT rules for that interface are still present, those rules are causing the NAT to stop working when the interface is disabled. If I disable those rules or re-enable the WAN interface NAT starts working again. This bug/issue started from CE v2.7.1. In CE v2.7.0 and previous versions, it was working fine.

ramikilany

Same things for me, I worked also with a network engineer and test a lot of things in our network up to the firewall. The internet connection is blocked in the LAN network, so we switched to other vlan from the core switch, it works for 5 minutes. after 5 minutes it block the internet connection try other vlan works and stopped working after a time. After more than 20 hours we have problem in the DHCP network the problem was only in the internet then it continue to not connect to the servers and between each others.

I downgraded to 2.7.0 and of course other issues happens:
1- pfblocker is not installed and cannot installed
2- the package manager stops showing the software to install (try to reinstall pfblocker)
3- the image disappear (not huge problem but just to mention)
4- i have crash in the system:

Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 RELENG_2_7_0-n255866-686c8d3c1f0: Wed Jun 28 04:21:19 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/obj/amd64/LwYAddCr/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/sources/FreeBSD-src-REL

Crash report details:

PHP Errors:
[21-Nov-2023 10:40:42 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'ftp.so' (tried: /usr/local/lib/php/20220829/ftp.so (Shared object "libssl.so.30" not found, required by "ftp.so"), /usr/local/lib/php/20220829/ftp.so.so (Cannot open "/usr/local/lib/php/20220829/ftp.so.so")) in Unknown on line 0
[21-Nov-2023 10:45:42 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'ftp.so' (tried: /usr/local/lib/php/20220829/ftp.so (Shared object "libssl.so.30" not found, required by "ftp.so"), /usr/local/lib/php/20220829/ftp.so.so (Cannot open "/usr/local/lib/php/20220829/ftp.so.so")) in Unknown on line 0
[21-Nov-2023 10:50:43 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'ftp.so' (tried: /usr/local/lib/php/20220829/ftp.so (Shared object "libssl.so.30" not found, required by "ftp.so"), /usr/local/lib/php/20220829/ftp.so.so (Cannot open "/usr/local/lib/php/20220829/ftp.so.so")) in Unknown on line 0
[21-Nov-2023 10:55:43 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'ftp.so' (tried: /usr/local/lib/php/20220829/ftp.so (Shared object "libssl.so.30" not found, required by "ftp.so"), /usr/local/lib/php/20220829/ftp.so.so (Cannot open "/usr/local/lib/php/20220829/ftp.so.so")) in Unknown on line 0
[21-Nov-2023 11:00:43 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'ftp.so' (tried: /usr/local/lib/php/20220829/ftp.so (Shared object "libssl.so.30" not found, required by "ftp.so"), /usr/local/lib/php/20220829/ftp.so.so (Cannot open "/usr/local/lib/php/20220829/ftp.so.so")) in Unknown on line 0
[21-Nov-2023 11:05:44 UTC] PHP Warning: PHP Startup: Unable to load dynamic library 'ftp.so' (tried: /usr/local/lib/php/20220829/ftp.so (Shared object "libssl.so.30" not found, required by "ftp.so"), /usr/local/lib/php/20220829/ftp.so.so (Cannot open "/usr/local/lib/php/20220829/ftp.so.so")) in Unknown on line 0

No FreeBSD crash data found
Screenshot from 2023-11-21 13-09-52.png

SteveITS

@ramikilany when you installed 2.7.0 did you change your update branch to Previous before trying to install any packages (from the later version)?

stephenw10

@coldfire7 said in No internet after upgrading to 2.7.1-ce from 2.7.0-ce:

I disabled number 2 a few days ago since the connection was offline due to a fibre cut but the NAT rules for that interface are still present, those rules are causing the NAT to stop working when the interface is disabled.

NAT rules can be present, they don't direct traffic. NAT rules only translate traffic that is already leaving that interface.

So you can see in the states opened by that ping the traffic leaving on the DOT interface but not being NAT'd. There are probably no NAT rules on DOT. Or at least none that match. NAT rule on the DHK interface have no effect there. What NAT rules do you have on DOT?

chris1284

@SteveITS said in No internet after upgrading to 2.7.1-ce from 2.7.0-ce:

@chris1284 re: DHCP, see if it’s this: https://redmine.pfsense.org/issues/15011

don't know, was an unacceptable situation so reinstall and config restore was a quick solution

coldfire7

@stephenw10 NAT rules are present for all the WAN and VPN interfaces.

I checked this like 10 times and I'm 100% sure. If I disable the WAN 2 (DHK) interface while the WAN 2 NAT rules are present, NAT stops working for all interfaces. I either have to re-enable the WAN 2 interface or remove/disable the WAN 2 NAT rules to get it back working again.

stephenw10

Hmm, hard to see how that could affect traffic on the DOT interface, which I assume is WAN1?

With WAN2 connected and all the NAT rules enabled is working traffic still using WAN1?

coldfire7

@stephenw10 said in No internet after upgrading to 2.7.1-ce from 2.7.0-ce:

Hmm, hard to see how that could affect traffic on the DOT interface, which I assume is WAN1?

Yeah, it doesn't make sense to me either. WAN1=DOT WAN2=DHK

@stephenw10 said in No internet after upgrading to 2.7.1-ce from 2.7.0-ce:

With WAN2 connected and all the NAT rules enabled is working traffic still using WAN1?

Yeah, as long as the interface is enabled, it doesn't have to be connected.

Also, note both WANs are PPPoE.

stephenw10

Ah, is it possible they share the same gateway address? The two PPPoE links I have hit that issue.

coldfire7

@stephenw10 Nope, totally different IP prefixes.

I noticed something else. The NAT address changes from DHK addresses to opt1ip when I disable the WAN 2 (DHK) interface, this doesn't happen in CE 2.7.0 or older versions.

SteveITS

@coldfire7 Well that sounds awfully suspicious. Does that part of the config file change when you make that change and redownload it?

avi68

I've had the same experience as you. I posted in the wrong section but the issues are exactly the same.

https://forum.netgate.com/topic/184213/openvpn-client-and-device-routing

coldfire7

@SteveITS

The only difference is <target></target> vs <target>opt1ip</target>. In CE 2.7.0 it's empty.

# 2.7.0

<rule>
	<source>
		<network>lan_v4</network>
	</source>
	<sourceport></sourceport>
	<descr><![CDATA[LAN --&gt; DHK]]></descr>
	<target></target>
	<targetip></targetip>
	<targetip_subnet></targetip_subnet>
	<interface>opt1</interface>
	<poolopts></poolopts>
	<source_hash_key></source_hash_key>
	<ipprotocol>inet</ipprotocol>
	<destination>
		<any></any>
	</destination>
	<created>
		<time>1592833956</time>
		<username><![CDATA[admin@10.0.0.100 (Local Database)]]></username>
	</created>
	<updated>
		<time>1676068535</time>
		<username><![CDATA[admin@10.0.0.100 (Local Database)]]></username>
	</updated>
</rule>

# 2.7.1 (DHK Interface Disabled)

<rule>
    <source>
        <network>lan_v4</network>
    </source>
    <sourceport></sourceport>
    <descr><![CDATA[LAN --&gt; DHK]]></descr>
    <target>opt1ip</target>
    <interface>opt1</interface>
    <poolopts></poolopts>
    <source_hash_key></source_hash_key>
    <ipprotocol>inet</ipprotocol>
    <destination>
        <any></any>
    </destination>
    <created>
        <time>1592833956</time>
        <username><![CDATA[admin@10.0.0.100 (Local Database)]]></username>
    </created>
    <updated>
        <time>1676068535</time>
        <username><![CDATA[admin@10.0.0.100 (Local Database)]]></username>
    </updated>
    <target_subnet></target_subnet>
</rule>

# 2.7.1 (DHK Interface Enabled)

<rule>
    <source>
        <network>lan_v4</network>
    </source>
    <sourceport></sourceport>
    <descr><![CDATA[LAN --&gt; DHK]]></descr>
    <target>opt1ip</target>
    <interface>opt1</interface>
    <poolopts></poolopts>
    <source_hash_key></source_hash_key>
    <ipprotocol>inet</ipprotocol>
    <destination>
        <any></any>
    </destination>
    <created>
        <time>1592833956</time>
        <username><![CDATA[admin@10.0.0.100 (Local Database)]]></username>
    </created>
    <updated>
        <time>1676068535</time>
        <username><![CDATA[admin@10.0.0.100 (Local Database)]]></username>
    </updated>
    <target_subnet></target_subnet>
</rule>

stephenw10

Hmm, interesting there was a change the system aliases available there.

How exactly are you disabling the interface?

stephenw10

Do you have any floating rules will all interfaces selected? Match rules for traffic shaping perhaps?

stephenw10

Struggling to replicate that here. If I disable a WAN interface then manual OBN rules on it become invalid and the ruleset shows:
# Missing interface 'opt1' for rule 'Test'

Automatic rules are simply not added for it.

OBN rules on the main WAN are still created and function as expected.

The OBN page does show opt1ip the same as you see but it doesn't impact anything.

I do also note the auto OBN rules do not show a translation address in the gui for some reason. But that also doesn't affect function.

SteveITS

@coldfire7 said in No internet after upgrading to 2.7.1-ce from 2.7.0-ce:

The only difference is <target></target> vs <target>opt1ip</target>. In CE 2.7.0 it's empty.

So it's <target></target> before you disable the interface? (I guess, I was asking for the difference between enabled/disabled, not 2.7.0 and 2.7.1)

stephenw10

It's <target>opt1ip</target> in 2.7.1 whether or not opt1 is enabled. The difference is that when opt1 is disabled opt1ip is not valid.

But in my test case the ruleset is still correctly generated. The only quirk is that the gui then shows the system alias. That probably shouldn't happen but it's only in the gui.

stephenw10

Ok it appears to be the OBN rules running together: https://redmine.pfsense.org/issues/15024

So here the rules for DHK are commented out when the DHK interface is disabled but ends up breaking the following rule which in each case is the DOT rule.

You should be able to see that in the generated rules.debug file.

coldfire7

@stephenw10 said in No internet after upgrading to 2.7.1-ce from 2.7.0-ce:

How exactly are you disabling the interface?

@stephenw10 said in No internet after upgrading to 2.7.1-ce from 2.7.0-ce:

Do you have any floating rules will all interfaces selected? Match rules for traffic shaping perhaps?

Yes, one for stopping outbound RFC1918. While I was troubleshooting yesterday I disabled that rule to see if that was causing the problem.