PfSense 2.3 LAN interface stops routing traffic - stops working after 2 or 3 day

j.koopmann

Happened again today twice but with different details.

LAN went down. For whatever reason my script doing ifconfig down and up did not help. I tried manually with no luck. Retried and introduced longer sleep between down and up and then it worked. This was the first crash in several days.
appr. 60 minutes later LAN was down again. However this time I did not even manage to get any result on the serial console. I had to power down/up. No core dump I was able to find however after login I saw a crash and uploaded it. not sure if it is related. I would say it happened around the time of (1).


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address	= 0x0
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80d22566
stack pointer	        = 0x28:0xfffffe001a38c590
frame pointer	        = 0x28:0xfffffe001a38c770
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 12 (irq260: igb2:que 0)
version.txt06000025412713111367  7616 ustarrootwheelFreeBSD 10.3-RELEASE #31 01118b4(RELENG_2_3): Thu Apr 28 03:57:55 CDT 2016
    root@ce23-amd64-builder:/builder/pfsense/tmp/obj/builder/pfsense/tmp/FreeBSD-src/sys/pfSense

Regards,
JP

jswope

I have 5 locations and we use the Dell R210 servers all sites except 2 have this issue. Only difference between them are Internet Service Providers.

Site 1 R210 Charter Fiber no issues

Site 2 AT&T LTE charter fiber bo issues

Site 3 Charter Fiber issues every 2-3 days watchdog error no lan routing

Site 4 charter coax internet issues every 3-days

marcvb

A other firewall of us got stuck just now, when i looked at the console it was showing a different wan adres.
It seems to have been reverted to a private, will take a screenshot next time.

cmb

We seem to have a fix for this. It's not yet merged into the source tree, but will be soon.

@jswope:

I have 5 locations and we use the Dell R210 servers all sites except 2 have this issue. Only difference between them are Internet Service Providers.

It's not universal or even close to it or we wouldn't have released with the issue. Not ISP-specific. Just happens the traffic profile of some systems will encounter it. Certain UDP streams is what triggers it.

afreaken

@cmb:

We seem to have a fix for this. It's not yet merged into the source tree, but will be soon.

@jswope:

I have 5 locations and we use the Dell R210 servers all sites except 2 have this issue. Only difference between them are Internet Service Providers.

It's not universal or even close to it or we wouldn't have released with the issue. Not ISP-specific. Just happens the traffic profile of some systems will encounter it. Certain UDP streams is what triggers it.

Do you have an ETA?

cmb

Sometime this week.

eeit

Hello,

any news on this ? Is there a special setting or a patch to fix the issue available ?

Thx.

cmb

Reducing the system to one core is the only workaround. The fix was merged into devel branch today and hopefully into RELENG_2_3 tomorrow, in which case it'll hit snapshots tomorrow.

jonnyaalgaard.org

Then I have upgraded to the new snapshot that came this morning (2.3.1-DEVELOPMENT (amd64) - built on Thu May 12 00:18:20 CDT 2016 - FreeBSD 10.3-RELEASE-p2), and will check it out.
I have also removed the temp workaround, from /boot/loader.conf.local
The temp workaround for me were to add these lines into loader.conf.local
hint.lapic.1.disabled=1
hint.lapic.2.disabled=1
hint.lapic.3.disabled=1

When I entered these lines pfsense were stabile for me, but now I have enabled all cpu cores again, and hopefully the new snapshot will work just fine.
Before the workaround I did have 10-12 “crashes” a day.
I will post back if the new snapshot works Ok or not.

UPDATE! Before i managed to post this my pfsense crashed again with the new snapshot. I will try to disable all but 1 cpu core again, still using todays snapshot.

cmb

There is no fix yet, as I noted. Watch the bug ticket for updates.

cmb

The fix for this was merged this morning and is in the most recent available snapshot, Thu May 12 14:01:47 CDT and newer.

My test setups are around 4.5 hours run time in a scenario that never lasted more than 3-4 hours without the fix. Another user who had a circumstance that was much faster to replicate than anything I could duplicate in a lab (a matter of a handful of minutes, rather than hours) has also confirmed it's no longer happening.

Needs more runtime and more feedback from others, but initial results are good.

Those of you impacted, please upgrade to latest 2.3.1 (instructions here), remove the disabling cores workaround if you did that, and let us know how it goes.

byusinger84

@cmb:

The fix for this was merged this morning and is in the most recent available snapshot, Thu May 12 14:01:47 CDT and newer.

My test setups are around 4.5 hours run time in a scenario that never lasted more than 3-4 hours without the fix. Another user who had a circumstance that was much faster to replicate than anything I could duplicate in a lab (a matter of a handful of minutes, rather than hours) has also confirmed it's no longer happening.

Needs more runtime and more feedback from others, but initial results are good.

Those of you impacted, please upgrade to latest 2.3.1 (instructions here), remove the disabling cores workaround if you did that, and let us know how it goes.

OP here. Testing this for you now. It usually crashed on me within 6-12 hours but up to 24 hours was the latest. I should know no later than this weekend if it is working or not. I'll update you when I know more. Thank you!

cmb

My test environments are now over 11.5 hours and still running fine. That's 7.5+ hours longer than any affected kernel has lasted in the circumstance. Definitely seems to be fixed.

Any feedback from those impacted who upgrade appreciated.

eeit

Hello,

guess 2.3.1.a.20160512.2347 works, but now i get permanent crash reports with informations about php-warnings like this:

10.3-RELEASE-p2
FreeBSD 10.3-RELEASE-p2 #68 ac020b1(RELENG_2_3): Fri May 13 00:26:15 CDT 2016 root@ce23-amd64-builder:/builder/pfsense/tmp/obj/builder/pfsense/tmp/FreeBSD-src/sys/pfSense

Crash report details:

PHP Errors:
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
[13-May-2016 04:24:10 America/New_York] PHP Warning: Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
[13-May-2016 04:24:10 America/New_York] PHP Stack trace:
[13-May-2016 04:24:10 America/New_York] PHP 1. {main}() /usr/local/www/firewall_rules.php:0
[13-May-2016 04:24:10 America/New_York] PHP 2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551

This informs are in the front of the Firewall -> Rules -> .. page too.

Any idea to fix this ?

Thx.

fragged

@eeit:

Hello,

guess 2.3.1.a.20160512.2347 works, but now i get permanent crash reports with informations about php-warnings like this:

Any idea to fix this ?

Thx.

Fixed in: https://github.com/pfsense/pfsense/commit/4680f6bf755fa7d323beba599ea94646d2d5f3bb

cmb

@fragged:

@eeit:

Hello,

guess 2.3.1.a.20160512.2347 works, but now i get permanent crash reports with informations about php-warnings like this:

Any idea to fix this ?

Thx.

Fixed in: https://github.com/pfsense/pfsense/commit/4680f6bf755fa7d323beba599ea94646d2d5f3bb

Yeah that was fixed. It's only cosmetic, for those running a version with that issue.

afreaken

@cmb:

The fix for this was merged this morning and is in the most recent available snapshot, Thu May 12 14:01:47 CDT and newer.

My test setups are around 4.5 hours run time in a scenario that never lasted more than 3-4 hours without the fix. Another user who had a circumstance that was much faster to replicate than anything I could duplicate in a lab (a matter of a handful of minutes, rather than hours) has also confirmed it's no longer happening.

Needs more runtime and more feedback from others, but initial results are good.

Those of you impacted, please upgrade to latest 2.3.1 (instructions here), remove the disabling cores workaround if you did that, and let us know how it goes.

So I followed the instructions on that page, and I get an error if I try to upgrade using the development option on the updates settings.

ERROR: Error trying to get packages list. Aborting…
pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/packagesite.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/packagesite.txz: Forbidden pkg: Repository pfSense-core cannot be opened. 'pkg update' required pkg: Repository pfSense cannot be opened. 'pkg update' required
ERROR: Error trying to get packages list. Aborting...
pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/packagesite.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/packagesite.txz: Forbidden pkg: Repository pfSense-core cannot be opened. 'pkg update' required pkg: Repository pfSense cannot be opened. 'pkg update' required

Also, afaik the option to upgrade from local file is no longer an option?
So unless I save it to a usb, and drive over to the other location, what can I do?

adam65535

Things are looking good so far according to all the posts by others. Nice job dev team! It looks like 2.3.1 is getting close to a final release now. Only issues set for 2.3.1 release are now in feedback status…

jimp

@afreaken:

So I followed the instructions on that page, and I get an error if I try to upgrade using the development option on the updates settings.

ERROR: Error trying to get packages list. Aborting…
pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/packagesite.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/packagesite.txz: Forbidden pkg: Repository pfSense-core cannot be opened. 'pkg update' required pkg: Repository pfSense cannot be opened. 'pkg update' required
ERROR: Error trying to get packages list. Aborting...
pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/packagesite.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/packagesite.txz: Forbidden pkg: Repository pfSense-core cannot be opened. 'pkg update' required pkg: Repository pfSense cannot be opened. 'pkg update' required

Looks like you're on the factory image, not CE, so the snapshots aren't available there currently. I'd suggest waiting for 2.3.1-RELEASE which if all goes as planned, will be first thing next week. You could, potentially, switch to CE snapshots but you'd lose the factory image tweaks/extras.

afreaken

@jimp:

Looks like you're on the factory image, not CE, so the snapshots aren't available there currently. I'd suggest waiting for 2.3.1-RELEASE which if all goes as planned, will be first thing next week. You could, potentially, switch to CE snapshots but you'd lose the factory image tweaks/extras.

Sure, that makes sense. Now I know why I don't see "Community Edition" on my UI, didn't realize there was a tweaked version on my system. In the past I've run the CE edition (obviously not on this machine).