Allowed memory size exhausted
-
Yeah, that's broken!
What exactly is this rogue data that's filling the file?
What reason is given for the config change in Diag > Backup&Restore > Config history?
Steve
-
Hi there!
I have exactly the same issue and I do hope we can figure this out to fix this problem / bug.
Our pfsense "crashed" last week and I could not access the GUI at all. It simply stated:
504 Gateway Time-out
nginxNot thinking much, I rebooted pfsense (which was a mistake as it turns out). After 10 minutes it was still not back online, so I attached a monitor to the gateway and started debugging this issue.
In the end I figured out that we had the exact same issue as stated here. I have no idea why this worked for so many years.
From one day to the next the config blew up and killed the setup (this was not after a pfsense upgrade or after changing anything in the webgui, I was not even in the office when this happened).Little more details:
On bootup it took nearly 45 minutes to get back "online" (which means no gui, but shell access). The config was around 60 MB at that time (also the backups were quite big).
On boot pfsense tried to parse the config and the backup which resulte in "Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in /etc/inc/xmlparse.inc on line 71".After checking here, I increased the php cache to 1024 MB and rebooted. This took - again - 45 minutes to complete. It did not show the error again, but it was still horribly slow.
After having a shell again, it turned out that /var/ was mounted as a 3,7 MB FS which ran full and showed "107% usage" (with df -h). I never had any issues with this before.
At this timepoint my config already had 130 MB.In the end I moved the config backups to a different dir and restored the first backup I had (which only had around ~100 KB). After a reboot pfsense was back up in around 5 minutes and started working perfectly fine again.
Just a note: I did not change anything about the /var/ FS, but after restoring the smaller config /var/ was not full anymore. I increased it afterwards to 100M just in case.Sadly today I noticed the same behavior again... Can't use the webgui, on logging in I can see that the config.xml grew again to over 50MB. This somehow stopped php from working correctly (/var/ is not full at the moment)
So please help me understand and fix this issue. I do not want to restore an old config every week ;-)
Interesting was the growth in config size:
ls -la total 453772 drwxr-xr-x 2 root wheel 1024 Aug 1 10:08 . drwxr-xr-x 5 root wheel 1024 Aug 1 12:52 .. -rw-r--r-- 1 root wheel 85933 Jul 18 13:44 config-1563450244.xml -rw-r--r-- 1 root wheel 103259 Jul 18 13:45 config-1563450245.xml -rw-r--r-- 1 root wheel 104373 Jul 18 13:45 config-1563450316.xml -rw-r--r-- 1 root wheel 139012 Jul 18 13:45 config-1563450318.xml -rw-r--r-- 1 root wheel 139027 Jul 18 13:45 config-1563450331.xml -rw-r--r-- 1 root wheel 208304 Jul 18 13:45 config-1563450333.xml -rw-r--r-- 1 root wheel 207217 Jul 18 13:45 config-1563450349.xml -rw-r--r-- 1 root wheel 207188 Jul 18 13:45 config-1563450356.xml -rw-r--r-- 1 root wheel 207189 Jul 18 13:48 config-1563450358.xml -rw-r--r-- 1 root wheel 207189 Jul 18 13:48 config-1563450492.xml -rw-r--r-- 1 root wheel 207188 Jul 18 13:48 config-1563450493.xml -rw-r--r-- 1 root wheel 345741 Jul 18 14:02 config-1563450495.xml -rw-r--r-- 1 root wheel 345682 Jul 18 14:03 config-1563451324.xml -rw-r--r-- 1 root wheel 622787 Jul 18 15:02 config-1563451388.xml -rw-r--r-- 1 root wheel 622786 Jul 18 15:03 config-1563454923.xml -rw-r--r-- 1 root wheel 1176995 Jul 18 16:02 config-1563454988.xml -rw-r--r-- 1 root wheel 1176994 Jul 18 16:03 config-1563458524.xml -rw-r--r-- 1 root wheel 2285411 Jul 18 17:02 config-1563458590.xml -rw-r--r-- 1 root wheel 2285410 Jul 18 17:03 config-1563462127.xml -rw-r--r-- 1 root wheel 4502243 Jul 18 18:02 config-1563462195.xml -rw-r--r-- 1 root wheel 4502242 Jul 18 18:03 config-1563465730.xml -rw-r--r-- 1 root wheel 8935907 Jul 18 19:02 config-1563465804.xml -rw-r--r-- 1 root wheel 8935906 Jul 18 19:03 config-1563469336.xml -rw-r--r-- 1 root wheel 17803235 Jul 18 20:02 config-1563469424.xml -rw-r--r-- 1 root wheel 17803234 Jul 18 20:04 config-1563472955.xml -rw-r--r-- 1 root wheel 35537891 Jul 18 21:03 config-1563473074.xml -rw-r--r-- 1 root wheel 35537890 Jul 31 19:02 config-1564590959.xml -rw-r--r-- 1 root wheel 35537890 Jul 31 19:04 config-1564592525.xml -rw-r--r-- 1 root wheel 71007203 Jul 31 20:00 config-1564592678.xml -rw-r--r-- 1 root wheel 71007202 Jul 31 20:10 config-1564596052.xml -rw-r--r-- 1 root wheel 141945827 Aug 1 10:08 config.xmlThis was the "first time"
And this is the second time:
ls -lah /conf/backup/ total 80632 drwxr-xr-x 2 root wheel 1.5K Aug 2 00:04 . drwxr-xr-x 4 root wheel 512B Aug 4 10:24 .. -rw-r--r-- 1 root wheel 4.7K Aug 2 00:04 backup.cache -rw-r--r-- 1 root wheel 135K Aug 1 12:35 config-1564655707.xml -rw-r--r-- 1 root wheel 202K Aug 1 12:38 config-1564655708.xml -rw-r--r-- 1 root wheel 203K Aug 1 12:38 config-1564655901.xml -rw-r--r-- 1 root wheel 203K Aug 1 12:39 config-1564655923.xml -rw-r--r-- 1 root wheel 203K Aug 1 12:40 config-1564655970.xml -rw-r--r-- 1 root wheel 202K Aug 1 12:40 config-1564656011.xml -rw-r--r-- 1 root wheel 202K Aug 1 12:44 config-1564656044.xml -rw-r--r-- 1 root wheel 202K Aug 1 12:44 config-1564656263.xml -rw-r--r-- 1 root wheel 202K Aug 1 12:44 config-1564656286.xml -rw-r--r-- 1 root wheel 202K Aug 1 12:52 config-1564656292.xml -rw-r--r-- 1 root wheel 202K Aug 1 12:53 config-1564656779.xml -rw-r--r-- 1 root wheel 202K Aug 1 12:53 config-1564656803.xml -rw-r--r-- 1 root wheel 201K Aug 1 12:53 config-1564656804.xml -rw-r--r-- 1 root wheel 330K Aug 1 12:53 config-1564656805.xml -rw-r--r-- 1 root wheel 328K Aug 1 12:54 config-1564656806.xml -rw-r--r-- 1 root wheel 328K Aug 1 12:54 config-1564656873.xml -rw-r--r-- 1 root wheel 329K Aug 1 12:54 config-1564656874.xml -rw-r--r-- 1 root wheel 602K Aug 1 12:54 config-1564656875.xml -rw-r--r-- 1 root wheel 603K Aug 1 12:54 config-1564656876.xml -rw-r--r-- 1 root wheel 605K Aug 1 12:54 config-1564656877.xml -rw-r--r-- 1 root wheel 1.1M Aug 1 12:54 config-1564656879.xml -rw-r--r-- 1 root wheel 1.1M Aug 1 12:55 config-1564656897.xml -rw-r--r-- 1 root wheel 1.1M Aug 1 12:58 config-1564656904.xml -rw-r--r-- 1 root wheel 1.1M Aug 1 13:00 config-1564657133.xml -rw-r--r-- 1 root wheel 1.1M Aug 1 13:00 config-1564657233.xml -rw-r--r-- 1 root wheel 1.1M Aug 1 13:00 config-1564657236.xml -rw-r--r-- 1 root wheel 2.2M Aug 1 14:00 config-1564657238.xml -rw-r--r-- 1 root wheel 4.3M Aug 1 15:00 config-1564660814.xml -rw-r--r-- 1 root wheel 8.5M Aug 1 16:00 config-1564664413.xml -rw-r--r-- 1 root wheel 17M Aug 1 17:01 config-1564668028.xml -rw-r--r-- 1 root wheel 34M Aug 2 00:04 config-1564671667.xmlAny idea why it starts growing the config file in one day this quickly?
If you need any information please let me know and I'll provide it.
Sadly I can currently not access the GUI so I cannot check the suggested "Diag > Backup&Restore > Config history"
But I do have access to the config files using SSH if this helps.
I could also restore an old config file, but I'm not sure what the config history will show then.Thank you in advance!
-
@waterstorm said in Allowed memory size exhausted:
Any idea why it starts growing the config file in one day this quickly?
No-one can do that. We have no access to your files.
You'll be needing text editor like Notepad++ - when installed, activate the Compare plugin.
You'll be needing a SFTP client (FileZilla will do).Now, compare two version of the config.xml - these are just text files.
-
@Gertjan sure
The <filter> part seems to be the issue.
In the "small" config it's from line 293 - 1327
While in the current version of the config the <filter> part is lines 298 - 1387151Some of the entries in the current config (if I search for a description) seem to be in the config 2048 times while they are only once in the small config.
Any specifics I should look out for? Any idea why the filter part blew up that quickly and got somehow replicated in the config (at least it looks like it)?
Thanks!
-
Are you running pfBlocker with auto rule re-ordering enabled?
We have seen that a few times though I'm not sure we found what triggers it:
https://redmine.pfsense.org/issues/8811Steve
-
Yes I'm running pfBlockerNG.
I was just checking the history and it is indeed created by pfBlockerNG. Thanks for pointing me in the right direction.This is interesting, because I have pfBlockerNG running forever and never had problems. However I did change something in pfBlockerNG lately. I needed whitelisting and therefore changed the "Rule Order" setting.
I just changed it back to the default to see if it fixes the issue. This was basically the only thing I changed in the settings of pfBlockerNG in the last year(s).
I'll report back if it fixes things. Thanks again. -
Yes, that's probably the cause. You will need to remove all the duplicated rules or role back the config to before that change.
As @BBcan177 (the author of pfBlocker-ng) said in that bug report try the development version of the pfBlocker package if you can and report back.
Steve
-
@stephenw10 Thank you!
I upgraded to the latest pfBlocker-ng-devel as recommended, so far everything works perfectly fine!
-
I just encountered this on a netgate device, SG-1100 which has only 1GB of RAM. I am troubleshooting now, but after uninstall of pfBlockerNG the device would not boot. I have removed the device from the route in order to debug more. Planning on doing a factory reset, which I assume will let it boot.
-
If you are able to reach the console menu and reset to factory defaults it will remove any spurious config, yes.
If you're able to reach single user mode you can mount the file system
mount -athen copy the default config from/conf.default/config.xmlto/conf/config.xmlthen reboot into it.Steve
-
The netgate device will not boot to any point where I can connect via LAN. the "console" port is a microUSB and does not work either. USB not recognized by systems. I will have to RMA the device.
-
Yes if you're unable to connect to the console port please open a ticket with us here: https://go.netgate.com
Steve
-
I went into the files described and increased the memory with an editor, saved the changes and resolved my memory issue. Good Stuff here. Thanks
-
P PTZ-M referenced this topic on
-
P PTZ-M referenced this topic on
-
S SteveITS referenced this topic on
-
S SteveITS referenced this topic on
-
S SteveITS referenced this topic on
-
S SteveITS referenced this topic on
-
S SteveITS referenced this topic on
-
S SteveITS referenced this topic on
