How to handle with big xml files and big posts
-
pfBlocker and upcoming postfix forwarder 2.3 can reach 128MB php limit very easy even on x64 instalation.
The error in these cases are:
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 57753 bytes) in /etc/inc/config.lib.inc on line 122
If I change memory limit on /etc/inc/config.inc I get "script tried to increase memory_limit alert".
I found on http://forum.pfsense.org/index.php?topic=30378.0 that I can increase 'suhosin.memory_limit' value, but even after running /etc/rc.php_ini_setup and reboot I get same error.
-
How are you hitting that limit? Is there not some way it could be redesigned so it does not hit that limit?
-
Basically two situations:
-
user submit a very large list to pfblocker(not a big deal. Until now i saw it twice)
-
upcoming postfix 2.3 has a sql inserts/updates maillogs forward via xmlrpc and depending on update schedule , host down or mail volume, xml file size reach 40-60mb
The xmlrpc sync was the best way I found to forward logs to another pfsense, so admins can search mail in only one box instead of looking on many pfsense smtp relays.
-
-
I've tested gzcompress php function and xml sql sync data reduced about 04-09 times.
I'll do more tests.
-
Check-out the xmlreader.inc it is very fast and can guarantee fixed memory consumption if used properly.
It is even faster than the parser used in pfSense and you can switch to it by a toggle.It is not used on 2.0 because there were some issue that iirc are fixed now.
