Allowed memory size exhausted

[RonpfS]

There is also a memory limit defined in /etc/inc/config.inc that you can raise :

// Set memory limit to 512M on amd64.
if ($ARCH == "amd64") {
	ini_set("memory_limit", "512M");
} else {
	ini_set("memory_limit", "128M");
}

This change will be overwritten on pfsense update or upgrade.

Save the config and open it in a editor to see what make it so large.

[sensemann]

I got it up and running again but now the size of config.xml is growing again!

2.4.4-RELEASE-p2 (amd64)
maschine:
16G RAM 128G SSD + Q550G6 6200U NO WiFi
Qotom-Mini-PC-with-Celeron-Core-i3-i5-Pfsense-AES-NI-6-Gigabit-NIC-Router

29M config.xml

[2.4.4-RELEASE][root@pfsense]/conf: du -sh backup/*
4.0K backup/backup.cache
800K backup/config-1555770468.xml
800K backup/config-1555770794.xml
800K backup/config-1555771468.xml
800K backup/config-1555771490.xml
1.2M backup/config-1555771510.xml
1.2M backup/config-1555771511.xml
2.1M backup/config-1555771514.xml
2.1M backup/config-1555771537.xml
2.1M backup/config-1555771542.xml
3.9M backup/config-1555771547.xml
3.9M backup/config-1555771595.xml
3.9M backup/config-1555771606.xml
3.9M backup/config-1555771607.xml
7.5M backup/config-1555771614.xml
7.5M backup/config-1555771622.xml
7.5M backup/config-1555771629.xml
7.5M backup/config-1555771631.xml
15M backup/config-1555771642.xml
15M backup/config-1555771930.xml
15M backup/config-1555772012.xml
15M backup/config-1555772061.xml
15M backup/config-1555772231.xml
15M backup/config-1555772442.xml

-rw-r--r-- 1 root wheel 3663 Apr 20 17:01 backup.cache
-rw-r--r-- 1 root wheel 765595 Apr 20 16:33 config-1555770468.xml
-rw-r--r-- 1 root wheel 765609 Apr 20 16:44 config-1555770794.xml
-rw-r--r-- 1 root wheel 765622 Apr 20 16:44 config-1555771468.xml
-rw-r--r-- 1 root wheel 766332 Apr 20 16:45 config-1555771490.xml
-rw-r--r-- 1 root wheel 1199572 Apr 20 16:45 config-1555771510.xml
-rw-r--r-- 1 root wheel 1200883 Apr 20 16:45 config-1555771511.xml
-rw-r--r-- 1 root wheel 2140855 Apr 20 16:45 config-1555771514.xml
-rw-r--r-- 1 root wheel 2140928 Apr 20 16:45 config-1555771537.xml
-rw-r--r-- 1 root wheel 2141470 Apr 20 16:45 config-1555771542.xml
-rw-r--r-- 1 root wheel 4021416 Apr 20 16:46 config-1555771547.xml
-rw-r--r-- 1 root wheel 4021489 Apr 20 16:46 config-1555771595.xml
-rw-r--r-- 1 root wheel 4021864 Apr 20 16:46 config-1555771606.xml
-rw-r--r-- 1 root wheel 4021829 Apr 20 16:46 config-1555771607.xml
-rw-r--r-- 1 root wheel 7781702 Apr 20 16:47 config-1555771614.xml
-rw-r--r-- 1 root wheel 7781775 Apr 20 16:47 config-1555771622.xml
-rw-r--r-- 1 root wheel 7781937 Apr 20 16:47 config-1555771629.xml
-rw-r--r-- 1 root wheel 7782320 Apr 20 16:47 config-1555771631.xml
-rw-r--r-- 1 root wheel 15302073 Apr 20 16:52 config-1555771642.xml
-rw-r--r-- 1 root wheel 15302119 Apr 20 16:53 config-1555771930.xml
-rw-r--r-- 1 root wheel 15302150 Apr 20 16:54 config-1555772012.xml
-rw-r--r-- 1 root wheel 15302235 Apr 20 16:57 config-1555772061.xml
-rw-r--r-- 1 root wheel 15302326 Apr 20 17:00 config-1555772231.xml
-rw-r--r-- 1 root wheel 15305296 Apr 20 17:01 config-1555772442.xml

Something is filling the config.xml
I cant find data parts like rrd etc... it looks like repeated configuration paramter
but </pfsense> is only on the end! one time.

In the 7.6MB file there is a rule (string like "nat_579a3e2d90a1b8.78260015" ) about 550 times in the textfile.

[stephenw10]

Yeah, that's broken!

What exactly is this rogue data that's filling the file?

What reason is given for the config change in Diag > Backup&Restore > Config history?

Steve

[waterstorm]

Hi there!

I have exactly the same issue and I do hope we can figure this out to fix this problem / bug.

Our pfsense "crashed" last week and I could not access the GUI at all. It simply stated:

504 Gateway Time-out
nginx

Not thinking much, I rebooted pfsense (which was a mistake as it turns out). After 10 minutes it was still not back online, so I attached a monitor to the gateway and started debugging this issue.

In the end I figured out that we had the exact same issue as stated here. I have no idea why this worked for so many years.
From one day to the next the config blew up and killed the setup (this was not after a pfsense upgrade or after changing anything in the webgui, I was not even in the office when this happened).

Little more details:
On bootup it took nearly 45 minutes to get back "online" (which means no gui, but shell access). The config was around 60 MB at that time (also the backups were quite big).
On boot pfsense tried to parse the config and the backup which resulte in "Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in /etc/inc/xmlparse.inc on line 71".

After checking here, I increased the php cache to 1024 MB and rebooted. This took - again - 45 minutes to complete. It did not show the error again, but it was still horribly slow.

After having a shell again, it turned out that /var/ was mounted as a 3,7 MB FS which ran full and showed "107% usage" (with df -h). I never had any issues with this before.
At this timepoint my config already had 130 MB.

In the end I moved the config backups to a different dir and restored the first backup I had (which only had around ~100 KB). After a reboot pfsense was back up in around 5 minutes and started working perfectly fine again.
Just a note: I did not change anything about the /var/ FS, but after restoring the smaller config /var/ was not full anymore. I increased it afterwards to 100M just in case.

Sadly today I noticed the same behavior again... Can't use the webgui, on logging in I can see that the config.xml grew again to over 50MB. This somehow stopped php from working correctly (/var/ is not full at the moment)

So please help me understand and fix this issue. I do not want to restore an old config every week ;-)

Interesting was the growth in config size:

ls -la
total 453772
drwxr-xr-x  2 root  wheel       1024 Aug  1 10:08 .
drwxr-xr-x  5 root  wheel       1024 Aug  1 12:52 ..
-rw-r--r--  1 root  wheel      85933 Jul 18 13:44 config-1563450244.xml
-rw-r--r--  1 root  wheel     103259 Jul 18 13:45 config-1563450245.xml
-rw-r--r--  1 root  wheel     104373 Jul 18 13:45 config-1563450316.xml
-rw-r--r--  1 root  wheel     139012 Jul 18 13:45 config-1563450318.xml
-rw-r--r--  1 root  wheel     139027 Jul 18 13:45 config-1563450331.xml
-rw-r--r--  1 root  wheel     208304 Jul 18 13:45 config-1563450333.xml
-rw-r--r--  1 root  wheel     207217 Jul 18 13:45 config-1563450349.xml
-rw-r--r--  1 root  wheel     207188 Jul 18 13:45 config-1563450356.xml
-rw-r--r--  1 root  wheel     207189 Jul 18 13:48 config-1563450358.xml
-rw-r--r--  1 root  wheel     207189 Jul 18 13:48 config-1563450492.xml
-rw-r--r--  1 root  wheel     207188 Jul 18 13:48 config-1563450493.xml
-rw-r--r--  1 root  wheel     345741 Jul 18 14:02 config-1563450495.xml
-rw-r--r--  1 root  wheel     345682 Jul 18 14:03 config-1563451324.xml
-rw-r--r--  1 root  wheel     622787 Jul 18 15:02 config-1563451388.xml
-rw-r--r--  1 root  wheel     622786 Jul 18 15:03 config-1563454923.xml
-rw-r--r--  1 root  wheel    1176995 Jul 18 16:02 config-1563454988.xml
-rw-r--r--  1 root  wheel    1176994 Jul 18 16:03 config-1563458524.xml
-rw-r--r--  1 root  wheel    2285411 Jul 18 17:02 config-1563458590.xml
-rw-r--r--  1 root  wheel    2285410 Jul 18 17:03 config-1563462127.xml
-rw-r--r--  1 root  wheel    4502243 Jul 18 18:02 config-1563462195.xml
-rw-r--r--  1 root  wheel    4502242 Jul 18 18:03 config-1563465730.xml
-rw-r--r--  1 root  wheel    8935907 Jul 18 19:02 config-1563465804.xml
-rw-r--r--  1 root  wheel    8935906 Jul 18 19:03 config-1563469336.xml
-rw-r--r--  1 root  wheel   17803235 Jul 18 20:02 config-1563469424.xml
-rw-r--r--  1 root  wheel   17803234 Jul 18 20:04 config-1563472955.xml
-rw-r--r--  1 root  wheel   35537891 Jul 18 21:03 config-1563473074.xml
-rw-r--r--  1 root  wheel   35537890 Jul 31 19:02 config-1564590959.xml
-rw-r--r--  1 root  wheel   35537890 Jul 31 19:04 config-1564592525.xml
-rw-r--r--  1 root  wheel   71007203 Jul 31 20:00 config-1564592678.xml
-rw-r--r--  1 root  wheel   71007202 Jul 31 20:10 config-1564596052.xml
-rw-r--r--  1 root  wheel  141945827 Aug  1 10:08 config.xml

This was the "first time"

And this is the second time:

ls -lah /conf/backup/
total 80632
drwxr-xr-x  2 root  wheel   1.5K Aug  2 00:04 .
drwxr-xr-x  4 root  wheel   512B Aug  4 10:24 ..
-rw-r--r--  1 root  wheel   4.7K Aug  2 00:04 backup.cache
-rw-r--r--  1 root  wheel   135K Aug  1 12:35 config-1564655707.xml
-rw-r--r--  1 root  wheel   202K Aug  1 12:38 config-1564655708.xml
-rw-r--r--  1 root  wheel   203K Aug  1 12:38 config-1564655901.xml
-rw-r--r--  1 root  wheel   203K Aug  1 12:39 config-1564655923.xml
-rw-r--r--  1 root  wheel   203K Aug  1 12:40 config-1564655970.xml
-rw-r--r--  1 root  wheel   202K Aug  1 12:40 config-1564656011.xml
-rw-r--r--  1 root  wheel   202K Aug  1 12:44 config-1564656044.xml
-rw-r--r--  1 root  wheel   202K Aug  1 12:44 config-1564656263.xml
-rw-r--r--  1 root  wheel   202K Aug  1 12:44 config-1564656286.xml
-rw-r--r--  1 root  wheel   202K Aug  1 12:52 config-1564656292.xml
-rw-r--r--  1 root  wheel   202K Aug  1 12:53 config-1564656779.xml
-rw-r--r--  1 root  wheel   202K Aug  1 12:53 config-1564656803.xml
-rw-r--r--  1 root  wheel   201K Aug  1 12:53 config-1564656804.xml
-rw-r--r--  1 root  wheel   330K Aug  1 12:53 config-1564656805.xml
-rw-r--r--  1 root  wheel   328K Aug  1 12:54 config-1564656806.xml
-rw-r--r--  1 root  wheel   328K Aug  1 12:54 config-1564656873.xml
-rw-r--r--  1 root  wheel   329K Aug  1 12:54 config-1564656874.xml
-rw-r--r--  1 root  wheel   602K Aug  1 12:54 config-1564656875.xml
-rw-r--r--  1 root  wheel   603K Aug  1 12:54 config-1564656876.xml
-rw-r--r--  1 root  wheel   605K Aug  1 12:54 config-1564656877.xml
-rw-r--r--  1 root  wheel   1.1M Aug  1 12:54 config-1564656879.xml
-rw-r--r--  1 root  wheel   1.1M Aug  1 12:55 config-1564656897.xml
-rw-r--r--  1 root  wheel   1.1M Aug  1 12:58 config-1564656904.xml
-rw-r--r--  1 root  wheel   1.1M Aug  1 13:00 config-1564657133.xml
-rw-r--r--  1 root  wheel   1.1M Aug  1 13:00 config-1564657233.xml
-rw-r--r--  1 root  wheel   1.1M Aug  1 13:00 config-1564657236.xml
-rw-r--r--  1 root  wheel   2.2M Aug  1 14:00 config-1564657238.xml
-rw-r--r--  1 root  wheel   4.3M Aug  1 15:00 config-1564660814.xml
-rw-r--r--  1 root  wheel   8.5M Aug  1 16:00 config-1564664413.xml
-rw-r--r--  1 root  wheel    17M Aug  1 17:01 config-1564668028.xml
-rw-r--r--  1 root  wheel    34M Aug  2 00:04 config-1564671667.xml

Any idea why it starts growing the config file in one day this quickly?

If you need any information please let me know and I'll provide it.

Sadly I can currently not access the GUI so I cannot check the suggested "Diag > Backup&Restore > Config history"

But I do have access to the config files using SSH if this helps.
I could also restore an old config file, but I'm not sure what the config history will show then.

Thank you in advance!

[Gertjan]

@waterstorm said in Allowed memory size exhausted:

Any idea why it starts growing the config file in one day this quickly?

No-one can do that. We have no access to your files.

You'll be needing text editor like Notepad++ - when installed, activate the Compare plugin.
You'll be needing a SFTP client (FileZilla will do).

Now, compare two version of the config.xml - these are just text files.

[waterstorm]

@Gertjan sure

The <filter> part seems to be the issue.

In the "small" config it's from line 293 - 1327
While in the current version of the config the <filter> part is lines 298 - 1387151

Some of the entries in the current config (if I search for a description) seem to be in the config 2048 times while they are only once in the small config.

Any specifics I should look out for? Any idea why the filter part blew up that quickly and got somehow replicated in the config (at least it looks like it)?

Thanks!

[stephenw10]

Are you running pfBlocker with auto rule re-ordering enabled?

We have seen that a few times though I'm not sure we found what triggers it:
https://redmine.pfsense.org/issues/8811

Steve

[waterstorm]

Yes I'm running pfBlockerNG.
I was just checking the history and it is indeed created by pfBlockerNG. Thanks for pointing me in the right direction.

This is interesting, because I have pfBlockerNG running forever and never had problems. However I did change something in pfBlockerNG lately. I needed whitelisting and therefore changed the "Rule Order" setting.

I just changed it back to the default to see if it fixes the issue. This was basically the only thing I changed in the settings of pfBlockerNG in the last year(s).
I'll report back if it fixes things. Thanks again.

[stephenw10]

Yes, that's probably the cause. You will need to remove all the duplicated rules or role back the config to before that change.

As @BBcan177 (the author of pfBlocker-ng) said in that bug report try the development version of the pfBlocker package if you can and report back.

Steve

[waterstorm]

@stephenw10 Thank you!

I upgraded to the latest pfBlocker-ng-devel as recommended, so far everything works perfectly fine!

[Kleinmann]

I just encountered this on a netgate device, SG-1100 which has only 1GB of RAM. I am troubleshooting now, but after uninstall of pfBlockerNG the device would not boot. I have removed the device from the route in order to debug more. Planning on doing a factory reset, which I assume will let it boot.

[stephenw10]

If you are able to reach the console menu and reset to factory defaults it will remove any spurious config, yes.

If you're able to reach single user mode you can mount the file system mount -a then copy the default config from /conf.default/config.xml to /conf/config.xml then reboot into it.

Steve

[Kleinmann]

The netgate device will not boot to any point where I can connect via LAN. the "console" port is a microUSB and does not work either. USB not recognized by systems. I will have to RMA the device.

[stephenw10]

Yes if you're unable to connect to the console port please open a ticket with us here: https://go.netgate.com

Steve

[ttime]

I went into the files described and increased the memory with an editor, saved the changes and resolved my memory issue. Good Stuff here. Thanks