PfSense 2.3 LAN interface stops routing traffic - stops working after 2 or 3 day
-
@cmb:
The fix for this was merged this morning and is in the most recent available snapshot, Thu May 12 14:01:47 CDT and newer.
My test setups are around 4.5 hours run time in a scenario that never lasted more than 3-4 hours without the fix. Another user who had a circumstance that was much faster to replicate than anything I could duplicate in a lab (a matter of a handful of minutes, rather than hours) has also confirmed it's no longer happening.
Needs more runtime and more feedback from others, but initial results are good.
Those of you impacted, please upgrade to latest 2.3.1 (instructions here), remove the disabling cores workaround if you did that, and let us know how it goes.
OP here. Testing this for you now. It usually crashed on me within 6-12 hours but up to 24 hours was the latest. I should know no later than this weekend if it is working or not. I'll update you when I know more. Thank you!
-
My test environments are now over 11.5 hours and still running fine. That's 7.5+ hours longer than any affected kernel has lasted in the circumstance. Definitely seems to be fixed.
Any feedback from those impacted who upgrade appreciated.
-
Hello,
guess 2.3.1.a.20160512.2347 works, but now i get permanent crash reports with informations about php-warnings like this:
10.3-RELEASE-p2
FreeBSD 10.3-RELEASE-p2 #68 ac020b1(RELENG_2_3): Fri May 13 00:26:15 CDT 2016 root@ce23-amd64-builder:/builder/pfsense/tmp/obj/builder/pfsense/tmp/FreeBSD-src/sys/pfSenseCrash report details:
PHP Errors:
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551This informs are in the front of the
Firewall -> Rules -> ..page too.Any idea to fix this ?
Thx.
-
Hello,
guess 2.3.1.a.20160512.2347 works, but now i get permanent crash reports with informations about php-warnings like this:
Any idea to fix this ?
Thx.
Fixed in: https://github.com/pfsense/pfsense/commit/4680f6bf755fa7d323beba599ea94646d2d5f3bb
-
Hello,
guess 2.3.1.a.20160512.2347 works, but now i get permanent crash reports with informations about php-warnings like this:
Any idea to fix this ?
Thx.
Fixed in: https://github.com/pfsense/pfsense/commit/4680f6bf755fa7d323beba599ea94646d2d5f3bb
Yeah that was fixed. It's only cosmetic, for those running a version with that issue.
-
@cmb:
The fix for this was merged this morning and is in the most recent available snapshot, Thu May 12 14:01:47 CDT and newer.
My test setups are around 4.5 hours run time in a scenario that never lasted more than 3-4 hours without the fix. Another user who had a circumstance that was much faster to replicate than anything I could duplicate in a lab (a matter of a handful of minutes, rather than hours) has also confirmed it's no longer happening.
Needs more runtime and more feedback from others, but initial results are good.
Those of you impacted, please upgrade to latest 2.3.1 (instructions here), remove the disabling cores workaround if you did that, and let us know how it goes.
So I followed the instructions on that page, and I get an error if I try to upgrade using the development option on the updates settings.
ERROR: Error trying to get packages list. Aborting…
pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/packagesite.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/packagesite.txz: Forbidden pkg: Repository pfSense-core cannot be opened. 'pkg update' required pkg: Repository pfSense cannot be opened. 'pkg update' required
ERROR: Error trying to get packages list. Aborting...
pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/packagesite.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/packagesite.txz: Forbidden pkg: Repository pfSense-core cannot be opened. 'pkg update' required pkg: Repository pfSense cannot be opened. 'pkg update' requiredAlso, afaik the option to upgrade from local file is no longer an option?
So unless I save it to a usb, and drive over to the other location, what can I do? -
Things are looking good so far according to all the posts by others. Nice job dev team! It looks like 2.3.1 is getting close to a final release now. Only issues set for 2.3.1 release are now in feedback status…
-
So I followed the instructions on that page, and I get an error if I try to upgrade using the development option on the updates settings.
ERROR: Error trying to get packages list. Aborting…
pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/packagesite.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/packagesite.txz: Forbidden pkg: Repository pfSense-core cannot be opened. 'pkg update' required pkg: Repository pfSense cannot be opened. 'pkg update' required
ERROR: Error trying to get packages list. Aborting...
pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/packagesite.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/packagesite.txz: Forbidden pkg: Repository pfSense-core cannot be opened. 'pkg update' required pkg: Repository pfSense cannot be opened. 'pkg update' requiredLooks like you're on the factory image, not CE, so the snapshots aren't available there currently. I'd suggest waiting for 2.3.1-RELEASE which if all goes as planned, will be first thing next week. You could, potentially, switch to CE snapshots but you'd lose the factory image tweaks/extras.
-
Looks like you're on the factory image, not CE, so the snapshots aren't available there currently. I'd suggest waiting for 2.3.1-RELEASE which if all goes as planned, will be first thing next week. You could, potentially, switch to CE snapshots but you'd lose the factory image tweaks/extras.
Sure, that makes sense. Now I know why I don't see "Community Edition" on my UI, didn't realize there was a tweaked version on my system. In the past I've run the CE edition (obviously not on this machine).
-
If/when 2.3.1 comes out next week, if I have already botched my Factory system by syncing up with 2.3.1CE is there any way to get back on the Factory track? Or if I do a backup/format/restore would that work?
-
If/when 2.3.1 comes out next week, if I have already botched my Factory system by syncing up with 2.3.1CE is there any way to get back on the Factory track? Or if I do a backup/format/restore would that work?
It's not quite that easy. It can be done with some file edits and such but it's not a very clean switch. Overall, less effort to reinstall+restore.
-
Overall, less effort to reinstall+restore.
ok so just to confirm, I can just pave, reinstall, and restore the config.xml – no extra tweaks or edits needed? thanks again
-
jimp, could you quickly comment on what caused the issue??
-
jimp, could you quickly comment on what caused the issue??
If you mean the original problem from this thread about the traffic stopping, I don't personally have that info. Should be on the tickets and/or in the commit history. It was something in the IPsec code, IIRC.
cmb is on a plane at the moment, he might have some more insight when he's able to respond.
-
We have the IPsec code from FreeBSD -CURRENT in 2.3.x. At the time we merged it in, there was an issue that wasn't identified and fixed until later. The fix was to re-merge all of IPsec from -CURRENT.
-
Looks like you're on the factory image, not CE, so the snapshots aren't available there currently. I'd suggest waiting for 2.3.1-RELEASE which if all goes as planned, will be first thing next week. You could, potentially, switch to CE snapshots but you'd lose the factory image tweaks/extras.
First thing next week, so is that today? Do you know if it is today?
-
The process has begun but we need time to test and such, so probably not today at this rate, but we'll see what happens.
The factory images gained access to snapshots late last week so you could run a snapshot now if you want, but with the release being so close, you may as well wait another day or so for it.
-
@cmb:
We have the IPsec code from FreeBSD -CURRENT in 2.3.x. At the time we merged it in, there was an issue that wasn't identified and fixed until later. The fix was to re-merge all of IPsec from -CURRENT.
OP checking in.
Looks like everything is good to go now. It's been up for over four days and not a single issue. Thanks for taking care of this! I anxiously await 2.3.1 so I can deploy to the rest of my sites!
-
The process has begun but we need time to test and such, so probably not today at this rate, but we'll see what happens.
The factory images gained access to snapshots late last week so you could run a snapshot now if you want, but with the release being so close, you may as well wait another day or so for it.
Well tried again for the factory image snapshot, however still getting an error when trying to get the update list. Hopefully 2.3.1 release will be out soon.
-
Hi,
running 2.3.1-DEVELOPMENT (i386) on net6501-70pfSense crashes EVERY time the remote IPsec user connects and attempts to access some Web pages on a local network.
SSH seems to be more stable. Submitted the crash report via Diagnostics/Crash reportsTried to disable one CPU - has no effect on the crash.
IPSec is not usable for us at this point …
