[2.8.1] PHP Fatal Error in filter.inc:2997 — fclose() on false when /var/etc/xinetd.conf cannot be created

hamsterplc

Hello,

CONTEXT

I've been dealing with a recurring issue (every 3 days or so) on pfSense 2.8.1 where a PHP fatal error in filter.inc causes NAT rules to stop being written, eventually breaking DNS and port forwards for all clients. The only issue put in place so far has been a full restore of the PfSense VM running on PVE 8.4.19.

Posting here in case anyone has seen this or can help narrow it down.

The error I get is:
PHP Fatal error: Uncaught TypeError: fclose(): Argument #1 ($stream)
must be of type resource, false given in /etc/inc/filter.inc:2997
The full stack trace points to filter_nat_rules_generate() called from filter_configure_sync().

DETAILS

Around line 2738, the function deletes /var/etc/xinetd.conf and immediately tries to recreate it with fopen(). If fopen() fails for any reason and returns false, the code doesn't check for that. It just carries on. Then at line 2997, fclose() is called unconditionally on what is now a false value, which throws the fatal error and stops execution before the NAT rules file is ever written.
The fix seems straightforward, fopen()'s return value should be validated before fclose() is called:
if ($xinetd_fd !== false) {
fclose($xinetd_fd);
}

SUSPECTED REASON

I suspect the fopen() failure is linked to /var being a 60MB tmpfs ramdisk. Over ~3 days it fills up progressively (we observed it climbing from 66% to 70% within minutes), and when it hits 100% during a network event that triggers a filter reload, fopen() can no longer create /var/etc/xinetd.conf returning false and triggering the crash.
This would also explain why I can reproduce the PHP error on demand by running:
php /etc/rc.filter_configure_sync 2>&1
...but NAT/DNS keeps working when I do because /var isn't full at that moment so pf already has a valid ruleset loaded. During the real crashes at night, something flushes the pf ruleset before calling filter_configure_sync(), and the fatal error leaves it empty.

NOTES

/var/etc/xinetd.conf is deleted then immediately recreated by the function — if creation fails, nothing downstream works
/var is a 60MB tmpfs, currently sitting at ~66-70% and actively growing
Environment: OVH VM, 6 VLANs, 4 OpenVPN servers, ZFS pool healthy
Crashes happen every ~3 days consistently, always overnight

QUESTIONS

Two open questions for the community:

Has anyone else hit this on 2.8.1, especia lly with a small /var ramdisk? Did changhing the size of the /var ramdisk help ?
What event would cause pf to flush its ruleset before calling filter_configure_sync — that seems to be the missing piece of why the real crashes are more severe than manual reproduction.

Thanks

Gertjan

@hamsterplc

@hamsterplc said in [2.8.1] PHP Fatal Error in filter.inc:2997 — fclose() on false when /var/etc/xinetd.conf cannot be created:

Around line 2738, the function deletes /var/etc/xinetd.conf and immediately tries to recreate it with fopen(). If fopen() fails for any reason and returns false, the code doesn't check for that. It just carries on. Then at line 2997, fclose() is called unconditionally on what is now a false value, which throws the fatal error and stops execution before the NAT rules file is ever written.
The fix seems straightforward, fopen()'s return value should be validated before fclose() is called:

I with you.

You've mentioned the real issue :

@hamsterplc said in [2.8.1] PHP Fatal Error in filter.inc:2997 — fclose() on false when /var/etc/xinetd.conf cannot be created:

Over ~3 days it fills up progressively (we observed it climbing from 66% to 70% within minutes)

find what or who is filling up this space, tell it to stop, and your issue will be gone.

Btw : the issue is very known : pfSense has some packages that make statistics from anything. The data is always stored in the /var/ folder. Most of the packages will auto prune the data, but if that fails, you pfSense will go down. Like any other device that doesn't have free disk space left.

stephenw10

Yup, the issue there is exhausting the space in /var. Yes, find out what is filling it.

Can you make the VAR ramdisk larger? 60MB is the absolute minimum size.

hamsterplc

@stephenw10 Thanks for your answer.

I did increase the size of /var to 512Mb.

@Gertjan Thanks for your answer.

I did seem the /var being autopruned as it usually went down to 65% when it reached 70%. I understand that a sudden increase in operation towards this folder would cause it reach 100% and causing a cascada of failures.

Would you have any suggestions on how to find what could be filling it ?

SteveITS

@hamsterplc "du -hd2 /var" should do it.