Serious errors on boot up and pfSense can't provide any connection

stephenw10

That error is showing you that PHP tried to use more memory than the process limit it set to; 512MB. Using that much almost always indicates some sort of error.

Because the menu command are mostly php many of them won't work. But you should be able to use option 8 to reach the command line.

First try just rebooting there by running reboot.

If it comes back up with the same error try running: /etc/rc.php-fpm_restart then: /etc/rc.restart_webgui

Steve

bthoven

@stephenw10 Thanks. I've just solved it. The config.xml was way too big, 52MB!. I replaced it with the backup one (500K) via usb drive copy. It is now working as usual.
I still don't know why the config file has grown so big. It could be somehow related to my recent upgrade from 2.6.x to 2.7.0 which ran fine for a few days after the upgrade. The backup config was produced just after the successful upgrade.

stephenw10

Huh, any idea what was in it making it that big? We have seen issues in the past where specific elements in the config were duplicated many times causes huge files. I'm not aware of anything current though.

Steve

bthoven

@stephenw10 As far as I can see, this is the rule section that has hundreds of duplications in the problematic config.xml. I didn't touch the Openvpn config at all before having the boot-up problem.

stephenw10

Hmm, curious. And this only happened some time after upgrading? Any idea what might have triggered it? Did you make any config changes?

bthoven

@stephenw10 After the upgrade, I got some errors notification on HAProxy-devel and it caused my Authelia (run as a docker in my separate Unraid box) not working properly. Some forum suggested it was a bug in that version of HAProxy-devel which latest version has resolved it; unfortunately it has not been updated on the pfSense package library. The only solution for now is to revert to the older production HAProxy package, which I did. The reason I was using the devel version was the HAProxy would stop itself as soon as my WAN ip changes. Let’s see whether running under the upgraded pfSense will cause the same problem.

After I revert back to HAProxy (non-devel) by removing the devel package and installing the non-devel one, no problem with pfSense. I then wanted to see whether going back to the devel version again will solve the errors which happens after the pfSense upgrade. During reinstalling the devel version, the installation took forever and I had to terminate it. Since then, pfSense started to have very slow response time while all other background things seems to be working as usual, but I can no longer access the Dashboard webpage. Restarting pfSense won’t solve the problem. I tried restarting pfSense several times and came to the worst one where nothing worked due to extra large size of the config file.

bthoven

@bthoven The problem was not permanently fixed. The file /var/run/check_reload_status keeps growing till 3.xMB, and the /cf/conf/config.xml keeps growing, now over 25MB. I did try to replace the backup config file 1.xMB once, and both files keep growing by themselves even though I didn't even log in to pfSense. The repeated config part in the config file is the OpenVPN Wizard as posted above. I don't know how to completely cure this problem.

Now I have to keep on deleting the check_reload_staus and replacing the config file with the backup one; otherwise, I believe, the webgui would be impossibly slow.

stephenw10

You mean it's that one firewall rule that gets duplicated every time? There is a config section for the OpenVPN Wizard specifically but I don't see that here.

Does it have the same tracker ID and creation timestamp as shown above? I.e. it's actually duplcating the rule rather then re-creating it.

How is check-reload-status expenading? Like memory usage shown in top? That could just be because the config is so much larger.

When you kill check-reload-status does the config stop getting larger until it is triggered again?

Steve

stephenw10

Are you running pfBlocker?

When you replaced the config did you reboot afterwards? If not did you remove the cached config, /tmp/config.cache ?

bthoven

@stephenw10 said in Serious errors on boot up and pfSense can't provide any connection:

Are you running pfBlocker?

When you replaced the config did you reboot afterwards? If not did you remove the cached config, /tmp/config.cache ?

I decided to replace the config with the one I backed up before upgrading to 2.7.0, and then reboot. So far so good.

bthoven

@bthoven It started to grow again from 818K (config file size just after the first reboot using a backup config) to, for now, 1.9MB.

After viewing the config file, I found there are thousands of the same rules inside. They all have the same tracker and time values. I sent part of the config file for your view here:
sample config file with duplication sections

The duplicated rule is this one:
```

                 <rule>
			<descr><![CDATA[OpenVPN Home Access OpenVPN wizard]]></descr>
			<source>
				<any></any>
			</source>
			<destination>
				<any></any>
			</destination>
			<interface>openvpn</interface>
			<type>pass</type>
			<enabled>on</enabled>
			<tracker>1563977113</tracker>
			<created>
				<time>1563977112</time>
				<username><![CDATA[OpenVPN Wizard]]></username>
			</created>
		</rule>

I also noticed that the 818K config file also has such duplication (same tracker number and timestamp). I believe when I replaced the config file with the backup one, size 520K, which worked perfectly on the previous 2.6.x version; rebooted pfSense; pfSense took some time to rebuild the system. After rebooting successfully, the config file size became 818KB which contains such duplication inside already. These duplications are seeds for further duplications if I have to change some config afterwards, thus it grows further. The webui still works, though the dashboard UI seems slower and slower to show up whilst other webpages are responsive as usual.

In summary, the duplication was initially the result of using the config backup of pfSense 2.6.x for booting up the 2.7.0 version.

I am not sure how to solve this problem.

stephenw10

So the config immediately went from 520k to 818k? Does it appear to add duplicates when you make changes? Or are duplicate rules added over time without any interaction?

Using a 2.6 config in 2.7 should be no problem.

Steve

bthoven

@stephenw10 It went from 520k to 818k immediately. It won’t add duplicates if I don’t make any changes.

Can I just delete the rule and all its duplicates, the reboot?

stephenw10

Yes, as long as you delete the xml parts correctly. Simply removing a rule(s) wont break the config.

bthoven

@stephenw10 I tried again today. This time I used my latest config backup in version 2.6, deleted the said rule and its duplicates (size 398KB), and then rebooted. It booted up properly with config size 399KB. I initially got no internet and later found my LAN interface rule, which allows any sources/ports was missing. Adding the rule back and the internet was back. I tried changing a few configurations and the config file has not grown like before.

I'll keep monitoring whether there will be any other side effects.

Thanks.

update on Oct 2, 2023: The problem has been permanently fixed, though I don't know how such Openvpn wizard creation xml section was there.