Issues with mail_reports.inc post upgrade to todays dev release

mtarbox

Not sure if anyone else has had an issue with mail_reports.
Just updated to 23.01.a.20221210.0318 and when I logged in, there was a crash report generated for it. The error pops up after every test email, so I guess reproducible and not a one off.

Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #0 plus-devel-main-n255994-d894b4aac11: Sat Dec 10 03:46:20 UTC 2022 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/obj/amd64/dC1e69BF/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBS

Crash report details:

PHP Errors:
[10-Dec-2022 10:11:09 America/New_York] PHP Fatal error: Uncaught Error: Failed opening required 'mail_reports.inc' (include_path='.:/etc/inc:/usr/local/pfSense/include:/usr/local/pfSense/include/www:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form:/usr/local/share/pear:/usr/local/share/openssl_x509_crl/') in /usr/local/bin/mail_reports_generate.php:24
Stack trace:
#0 {main}
thrown in /usr/local/bin/mail_reports_generate.php on line 24

No FreeBSD crash data found.

jimp

Remove/reinstall the package and see if it continues to happen. Seems like somehow that file it wants has gone missing.

mtarbox

@jimp
Removed, rebooted and reinstalled.
Still the same issue.
Also does it with today's release(23.01.a.20221212.1321), so something is persistent.
Of note, it doesn't send the email either, but if I reboot the unit, I get those reports(I know, different)
Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #0 plus-devel-main-n255994-d894b4aac11: Sat Dec 10 03:46:20 UTC 2022 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/obj/amd64/dC1e69BF/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBS

Crash report details:

PHP Errors:
[12-Dec-2022 16:03:55 America/New_York] PHP Fatal error: Uncaught Error: Failed opening required 'mail_reports.inc' (include_path='.:/etc/inc:/usr/local/pfSense/include:/usr/local/pfSense/include/www:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form:/usr/local/share/pear:/usr/local/share/openssl_x509_crl/') in /usr/local/bin/mail_reports_generate.php:24
Stack trace:
#0 {main}
thrown in /usr/local/bin/mail_reports_generate.php on line 24

No FreeBSD crash data found.

mtarbox

@jimp
Same results with tonight's release, 23.01.a.20221212.2233.
Removed mail reports, rebooted, and reinstalled.

mtarbox

Re: Issues with mail_reports.inc post upgrade to todays dev release

And these are the emails it sends:

X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>

Fatal error: Uncaught Error: Failed opening required 'mail_reports.inc' (include_path='.:/etc/inc:/usr/local/pfSense/include:/usr/local/pfSense/include/www:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form:/usr/local/share/pear:/usr/local/share/openssl_x509_crl/') in /usr/local/bin/mail_reports_generate.php:24
Stack trace:
#0 {main}
thrown in /usr/local/bin/mail_reports_generate.php on line 24

And then about three hours later I receive this:

X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin>
X-Cron-Env: <LOGNAME=root>
X-Cron-Env: <USER=root>

eval: mail: not found
cat: stdout: Broken pipe
cat: stdout: Broken pipe
cat: stdout: Broken pipe
cat: stdout: Broken pipe
cat: stdout: Broken pipe
cat: stdout: Broken pipe
cat: stdout: Broken pipe
cat: stdout: Broken pipe
cat: stdout: Broken pipe
echo: write error on stdout
echo: write error on stdout

mtarbox

Now I know I'm effed!
It still sends me emails, even with the package removed!
It's borked. At least it is my home unit, and not my production unit.

yuryk

I have exactly the same issue. It would help to know where this .inc should reside.

mtarbox

@yuryk because this is for a home lab, I’m just going to reformat and reinstall. I needed to update the bios on the fitlet2 anyway, so a perfect time. Flashing 2.6 now as I type this.

mtarbox

@jimp
And I just went through the process of re-installing 2.6, and the upgrade process to 22.01+ through to 23.01-BETA. Same issue.
No packages other than mail reports installed. As vanilla as I can get.

Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #0 plus-devel-main-n255994-d894b4aac11: Fri Dec 16 06:30:06 UTC 2022 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/obj/amd64/5QbEYaBj/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBS

Crash report details:

PHP Errors:
[16-Dec-2022 14:05:59 America/New_York] PHP Fatal error: Uncaught Error: Failed opening required 'mail_reports.inc' (include_path='.:/etc/inc:/usr/local/pfSense/include:/usr/local/pfSense/include/www:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form:/usr/local/share/pear:/usr/local/share/openssl_x509_crl/') in /usr/local/bin/mail_reports_generate.php:24
Stack trace:
#0 {main}
thrown in /usr/local/bin/mail_reports_generate.php on line 24
[16-Dec-2022 14:06:47 America/New_York] PHP Fatal error: Uncaught Error: Failed opening required 'mail_reports.inc' (include_path='.:/etc/inc:/usr/local/pfSense/include:/usr/local/pfSense/include/www:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form:/usr/local/share/pear:/usr/local/share/openssl_x509_crl/') in /usr/local/bin/mail_reports_generate.php:24
Stack trace:
#0 {main}
thrown in /usr/local/bin/mail_reports_generate.php on line 24
[16-Dec-2022 14:07:19 America/New_York] PHP Fatal error: Uncaught Error: Failed opening required 'mail_reports.inc' (include_path='.:/etc/inc:/usr/local/pfSense/include:/usr/local/pfSense/include/www:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form:/usr/local/share/pear:/usr/local/share/openssl_x509_crl/') in /usr/local/bin/mail_reports_generate.php:24
Stack trace:
#0 {main}
thrown in /usr/local/bin/mail_reports_generate.php on line 24
[16-Dec-2022 14:08:24 America/New_York] PHP Fatal error: Uncaught Error: Failed opening required 'mail_reports.inc' (include_path='.:/etc/inc:/usr/local/pfSense/include:/usr/local/pfSense/include/www:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form:/usr/local/share/pear:/usr/local/share/openssl_x509_crl/') in /usr/local/bin/mail_reports_generate.php:24
Stack trace:
#0 {main}
thrown in /usr/local/bin/mail_reports_generate.php on line 24

No FreeBSD crash data found.

yuryk

So this is what I did:
Looks like the include file was not included in the build...

Download a copy of the missing mail_report.inc from github and copy it into /etc/inc of the pfsense. That should fix the issue....

mtarbox

@yuryk YOU FREAKING ROCKSTAR! That fixed the issue!
@jimp see the above for the fix.

jimp

Copying that file is the wrong fix. There is just one line that needs changed in one file to have it load the correct include path/name.

https://redmine.pfsense.org/issues/13773

https://github.com/pfsense/FreeBSD-ports/commit/9fe472d465f90fb272ae9b4edb3f1598a904c1f3

mtarbox

@jimp Thank you. I'll wait until tomorrow, tickle the system for an update and go from there.