Fails to load after Jun 15 update

foonus

Thanks Bob :D

foonus

This is getting rather frustrating… 14 days later and still no update or information regarding this situation.
You'd think an issue like this preventing users from running pfSense at all would take some form of priority, but its almost half a month since this has sprung and not even a hint of anybody looking into this despite the documentation presented here as well as the direction of other posters indeed identifying that something changed in the software to cause this.

Don't know what more can be done to document this so that it gets resolved so I will put it rather blunt...

Users will not be able to install pfSense after June 12-13 on a HP Proliant DL320 (and possibly other DL36X) rack mount servers with internal NICs disabled until this is resolved, this is not a small percentage of users, many seek out this rack mount hardware specifically for firewalls.

Minor tweaks to pfSense are useless to users if they cant install it… let's get the major stuff out of the way first. This issue was introduced with a snapshot after June 12-13'th, I don't understand the reluctance in going back and actually looking at what changed to cause this and progressing from there. Somebody make the change, therefore somebody should know what they have done....

Forgive the frustration but aside from a few helpful users all of this documentation has fallen on deaf ears so far as the developers go, not a single one has commented in over half a month; which leads me to believe that they have not seen this thread (unlikely) or believe that this is an isolated issue and are not taking it seriously. Ermal,Rbgarga,Jim-p,Phil-davis... anyone seeing this???

jimp

Looks like someone updated the bge driver between those snapshots.

https://github.com/pfsense/pfsense-tools/commit/4e086e600e71daec708c37c45b265226c5d45186

I'm not sure if that commit actually helped anyone, but if it did create a regression we can back it out.

jimp

I just deactivated the patches from that commit, the next new snapshot that builds should have the previous driver again.

foonus

@jimp:

I just deactivated the patches from that commit, the next new snapshot that builds should have the previous driver again.

Thanks Jim, really appreciate it. Will test this immediately when the snapshot becomes available.

wallabybob

@foonus:

Users will not be able to install pfSense after June 12-13 on a HP Proliant DL320 (and possibly other DL36X) rack mount servers with internal NICs disabled until this is resolved, this is not a small percentage of users, many seek out this rack mount hardware specifically for firewalls.

Of those who seek out this class of hardware, how many would disable the internal NICs? And why?

foonus

@wallabybob:

@foonus:

Users will not be able to install pfSense after June 12-13 on a HP Proliant DL320 (and possibly other DL36X) rack mount servers with internal NICs disabled until this is resolved, this is not a small percentage of users, many seek out this rack mount hardware specifically for firewalls.

Of those who seek out this class of hardware, how many would disable the internal NICs? And why?

The machines are several years old now, can get the best of both worlds with server stability and today's current nics ;) Firmware updates are also much easier to do on your NIC when they are not packaged into an HP or Dell Firmware package! In the Gov. rollout here with the Proliants when we installed the servers we actually swapped out even the hardware nics out for Intels! The onboard ones were dissabled for sure! Some of the later gen DL320's actually come with a version of this 4 port NIC (ET or VT) from Intel directly installed in the machine.

@jimp:

I just deactivated the patches from that commit, the next new snapshot that builds should have the previous driver again.

You fixed it!
Shame they didn't update the Intel drivers instead of the Broadcom ones! ;D

jimp

@foonus:

@jimp:

I just deactivated the patches from that commit, the next new snapshot that builds should have the previous driver again.

You fixed it!
Shame they didn't update the Intel drivers instead of the Broadcom ones! ;D

We did. :-)

foonus

@jimp:

@foonus:

@jimp:

I just deactivated the patches from that commit, the next new snapshot that builds should have the previous driver again.

You fixed it!
Shame they didn't update the Intel drivers instead of the Broadcom ones! ;D

We did. :-)

You have no idea how much better the firewall runs now with these Intel drivers updated. I would say that you can literally SEE a difference. Now the line keeps its maximum 20Mbit upload saturated almost full time, before this update 3/4 saturation was the usual top end; and ya… it boots! I knew the Intel drivers were outdated, but I never considered that it would impact performance THAT much!
Thanks again, and well done!

jimp

Unfortunately, we might have to pull them back out. Apparently they break altq with igb. :(

foonus

Forgive my ignorance, but why would newer drivers be causing so much trouble in pfSense?
This whole issue was brought about by updating a Broadcomm driver, and now from what you are saying the Intel updated driver causes issues for some people as well.

Are both these drivers not released for FreeBSD by both companies? Isn't it their job to ensure that these things are tested before the drivers are released? Is this a quality control issue, or simply the fact that they are testing them on a kernel newer than what pfSense is running?

Rather disappointing that these large companies can release drivers in this state when they have people paid to maintain quality control, and the grief falls on the users and developers of this project that aren't being paid to deal with it.

jimp

@foonus:

Forgive my ignorance, but why would newer drivers be causing so much trouble in pfSense?
This whole issue was brought about by updating a Broadcomm driver, and now from what you are saying the Intel updated driver causes issues for some people as well.

Are both these drivers not released for FreeBSD by both companies? Isn't it their job to ensure that these things are tested before the drivers are released? Is this a quality control issue, or simply the fact that they are testing them on a kernel newer than what pfSense is running?

Rather disappointing that these large companies can release drivers in this state when they have people paid to maintain quality control, and the grief falls on the users and developers of this project that aren't being paid to deal with it.

Well we are on FreeBSD 8.3, the newer drivers are from 8-STABLE or HEAD/CURRENT. Sometimes the drivers depend on other features that are found there, but that we do not yet have. Usually they work fine, but on occasion there are changes to other subsystems that cause more subtle breakage.

foonus

In my case i am lucky since its running well now, will likely be using this snapshot until one is released with newer (fixed& > 2.3.9) Intel drivers due to the performance impact I have seen.

pfSense loads (now): <intel(r) 8="" 1000="" pro="" network="" connection="" version="" -="" 2.3.9="">- Works great for my setup - noticeable performance increase from 2.3.1
pfSense loads (before): <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.3.1="">- Over 2 years old, no longer supported by Intel!

Interesting that the Intel website shows the following for drivers:
2.3.8: This driver has been developed for FreeBSD 9.x kernel
2.3.7: This driver has been developed for use with FreeBSD 7.x kernel

7.x straight to 9.x right from Intel with no 8.3 in between. Also leads me to believe that the original 2.3.1 was for FreeBSD 7.x or prior.

What a mess.</intel(r)></intel(r)>

jimp

@foonus - Do you use traffic shaping? Have you tried using the shaper wizard with those igb drivers?

foonus

Luckily I don't run traffic shaping in this location, thus the reason I have made a hard copy of this snapshot with the new drivers!
I ran a test for you to confirm that there was a problem, simply ran the wizard to create it and it came back with:

There were error(s) loading the rules: pfctl: igb1: driver does not support altq - The line in question reads [ 0]:…

If you need any more done, let me know, will be glad to help.

jimp

There may not be much more we can do for now. I deactivated that last igb/em update. We could maybe compile a module that could be used/loaded as an alternate driver using that code.

Once we get 2.1 done, we'll be focusing on 2.2 on FreeBSD 10 which is leaps and bounds newer than what we have, driver-wise, all around.

athoik

Hello,

Same issue occurs with 2.1.RC0 (on vmware esxi 5.1).

The issue happens only on Pfsense17 with memory 6G. All others instances have 2G memory and no problem yet.

crash.png_thumb

athurdent

This could be totally unrelated, but I noticed the process bsnmpd there. I remember having an issue some time ago when using more that one virtual CPU/Core and collecting CPU info via SNMP (using Cacti). Turning off the CPU check or reducing the VM to only one CPU stopped the crashes.

msi

argh, yes they were backed out - but reason is valid:
https://github.com/pfsense/pfsense-tools/commit/0d63350445dad1e7281ba085cd9fa0e0fdd7e261