HP microserver N54L - have someone any problem if stress pfsense box?

marian78

Hi,

i have HP microserver N54L with installed pfsense v2.2.1 x64 on SSD disk (TRIM enabled).
My problem is, that if i download for example SUSE iso image with torrent on lan client, pfsense restart (too many connection or speed. CPU get max 70%, temp max 58C). I look at system log, but cant find any relevant message.

about HP microserver N54L:
CPU AMD Turion II Neo N54L @2,2GHz
RAM - 2x2G/1333 ECC unbuffered HP brand
SSD - 32GB
LAN1,2 - dual LAN PCIe 4x card Intel HP NC360T

Can you have same problem, if you stress pfsense box? How can i fing some logs, where can i find some info?

services.PNG_thumb

My problem is, that if i download for example SUSE iso image with torrent on lan client, pfsense restart (too many connection or speed. CPU get max 70%, temp max 58C).

Can it be that you have to many features, options and services activated and then the CPU
is going out of power to handle all this?

With another HP Proliant you could perhaps trying to change the CPU to a Intel Xeon E3-12xxv3
4 Core and upgrading the entire RAM also, but here in your tower it isn´t really going as I see it
right. Perhaps you will try out the following.

Disable most of all services
Try downloading again OpenSUSE
And then turn on step by step the services back so that you will see how many load the CPU
is able to handle, with witch services enabled.

cmb

You almost certainly have a hardware problem that gets hit when putting the system under higher load. Often that's bad RAM, or some other hardware issue causing memory corruption, where increased load uses somewhat more memory and makes it more likely it'll hit a bad spot.

Do you get prompted to submit a crash report when you get back into the web interface after it reboots?

marian78

first thx all for time.

When i got back, i have not any prompt to submit a crash report.
I also looked to /var/log/…, but there is nothing about why it crashed. Strange is, that microserver do not reboot, only pfsense UI is inaccesible for 1 minute while lost network connections.
I try replace RAM (again), with another modules.

Is there any log, where can find something about hw failures?

edwardwong

I think I can tell what's wrong, because I also owned HP N54L + Intel Pro 1000 PT Dual Port NIC (HP NC360T is actually Intel 82571EB)

First of all, I didn't install pfSense on my N54L, instead I installed FreeNAS on it (which is also a FreeBSD based distro), I can observe that when there is huge traffic on the dual port NIC, the machine will REBOOT silently!! When I transfer large file on LAN, it just needs < 10s to trigger reboot.

Try to disable some hardware feature like TCP Offload Engine (TSO4), and do some tweak following this doc.

In my case, problem wasn't resolved, but at least it won't be rebooted within seconds during on load (sometimes might be 1min) I have another Pro 1000 PT Dual Port (probably different manufacturing batches), swapped in and everything fine.

But I can tell the original card is not a failed card, because there isn't any issue on Linux/Windows, when I moved the card to another desktop PC with FreeBSD it works well! I just have no idea why N54L + 1st Intel Pro 1000 PT Dual port doesn't work under FreeBSD (but my 2nd card does working well). Maybe you'll have to recompile some driver modules? Not so sure….

Hope these info helps

cmb

I thought you meant it was randomly rebooting, if it's not, then it's not likely a general hardware problem with the system.

@edwardwong:

In my case, problem wasn't resolved, but at least it won't be rebooted within seconds during on load (sometimes might be 1min) I have another Pro 1000 PT Dual Port (probably different manufacturing batches), swapped in and everything fine.

But I can tell the original card is not a failed card, because there isn't any issue on Linux/Windows, when I moved the card to another desktop PC with FreeBSD it works well! I just have no idea why N54L + 1st Intel Pro 1000 PT Dual port doesn't work under FreeBSD (but my 2nd card does working well). Maybe you'll have to recompile some driver modules? Not so sure….

Thanks for the feedback! There is no way to recompile drivers, what's there is the best available Intel NIC driver.

Based on what edwardwong said about that system, I suspect it's probably a BIOS bug or setting of some sort. Make sure you're on the latest BIOS version, and reset the BIOS to factory defaults in case it had something weird set in there from previous use. Sometimes Windows and Linux have workarounds for hardware problems that FreeBSD doesn't have, and on occasion there are hardware bugs that Windows and Linux don't trigger that FreeBSD does.

edwardwong

In my case, it's not random rebooting, reboot only occurs when my Intel NIC is having high traffic passing thru.
HP N54L does have on board Broadcom NetXtreme GbE, I have no trouble using it to transfer file at all.

Even for a newer Microserver, Gen8, HP does have BIOS issue. For Gen8 with stock BIOS, people will experience machine hang up if they use Windows Server 2012 R2, which is very stupid. And for N54L, if you don't mod BIOS, your server will lost HDD hot plug + no multiplier support (but hardware is capable to handle this), so you are right, this could be BIOS problem.

But it could be Intel's problem as well, if you search on internet, there are plenty of forum posts talking about various reboot/hang up issue when using Intel NIC (especially 2-4 port model) with FreeBSD.

marian78

Hi,

now im in work, but i try remotely setup some changes in loader.conf.local and system tunable (https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards).

Next i put opensuse iso torrent to download. For now, no reboot. we will see…..

I will post result....

PS: onboard network nic pfsense 2.2.1 dont see.... :'( I have another H54L box with freenas 9.3 installed and i use onboard nic without any problem.

riahc3

It sounds too me like a HW problem. I have a N54L as well and I have no issues like this at all.

I think I even have less RAM than you.

Run MemTest on that system.

edwardwong

It has nothing to do with RAM, otherwise it won't work fine under Linux/Windows.
At home I have 6 sticks of ECC RAM for N54L, I also tried to test with each stick but no luck.

@edwardwong

But it could be Intel's problem as well, if you search on internet, there are plenty of forum posts talking about various reboot/hang up issue when using Intel NIC (especially 2-4 port model) with FreeBSD.

Intel i354 and i350 adapters should be flashed with the latest firmware and then the problems are gone.

onboard network nic pfsense 2.2.1 dont see…. :'( I have another H54L box with freenas 9.3 installed and i use onboard nic without any problem.

The I really think in FreeNAS is a driver inside that supports the NIC and in pfSense 2.2.1 this
driver is not inside. Did you try out an older version of pfSense?

marian78

Intel i354 and i350 adapters should be flashed

hm, must look to HP web support if they have new firmware (i have HP brand NIC).

The I really think in FreeNAS is a driver inside that supports the NIC and in pfSense 2.2.1 this
driver is not inside. Did you try out an older version of pfSense?

yes, before 2.2.1 i run pfsense from version 2.1.1 and onboard nic is working without problem. After fresh install version 2.2 onboard nic is not working

EDIT: ok, search for firmware on HP pages, but they dont release any for NC360T (or i cant find). Can you post, pleas some links?

EDIT2: onboard NIC is NC107i (BCM5723)

edwardwong

@marian78:

Hi,

now im in work, but i try remotely setup some changes in loader.conf.local and system tunable (https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards).

Next i put opensuse iso torrent to download. For now, no reboot. we will see…..

I will post result....

PS: onboard network nic pfsense 2.2.1 dont see.... :'( I have another H54L box with freenas 9.3 installed and i use onboard nic without any problem.

It's strange, pfSense 2.2.1 has a newer FreeBSD version then FreeNAS 9.3, but pfSense can't see the NIC…..

marian78

ok, now im at home. pfsense reboot again - i open on client pc in browser 20 more tabs with different pages and pfsense reboot.

If it is caused by intel nic, can i swap this card for example with two realtek cards without reinstalation of pfsense?

PS: onboard nic - yes it strange that pfsense 2.2.1 cant see it.

yes, before 2.2.1 i run pfsense from version 2.1.1 and onboard nic is working without problem. After fresh install version 2.2 onboard nic is not working

So as I see it right the only different is then FreeBSD 9.3 and FreeBSD 10.1 or am I wrogn with this?

EDIT: ok, search for firmware on HP pages, but they dont release any for NC360T (or i cant find). Can you post, pleas some links?

I really think this would not march! I was findign and reading some threads here in the forum
related even to the support of Intel i350 and i354 based NICs, either with 1, 2 or 4 LAN Ports!
And in some of the last threads there is often talked about to get a utility that can flash a new
firmware on the card and then all problems together with pfSense are gone! So I think it will be
for Intel original adapters only. Here are some links to this threads if you have interest on reading:

Hm, what should I say, I have read more threads about this Intel i350, i354 AM4 adapters I was
thinking but now I really found this some to show you up! Really sad but the entire sound in the back
was that pfsense 2.1.5 - 2.2.1 was really getting new drivers to solve the problem from there sides
and on the other side the firmware must in some rarely cases flashed to the latest one.
THE PROBLEM IS SOLVED. I TRIED TO USE Ethernet Flash Firmware Utility (BootUtil) TO UPDATE THE FIRMWARE OF THE I350-T4. FINALLY GET LUCKY. IT WORKS NOW!

It's strange, pfSense 2.2.1 has a newer FreeBSD version then FreeNAS 9.3, but pfSense can't see the NIC…..

Perhaps not all drivers was finding its way into the new release? Can this be? Perhaps they sorted out
some things not to blow the entire Release could this be? Ok for sure at one time, many distributions
are sorting out drivers for older hardware or go straight a new way where more some hardware from
each section will be supported. Ok it ca easily surround by offering DVD images from FreeBAS or
pfSense, perhaps some peoples from staff or jumping in and answering something on that.

cmb

@BlueKobold:

Perhaps not all drivers was finding its way into the new release? Can this be?

No, all the drivers are there.

There are a small number of instances where that Intel firmware update is needed which fixed this. That seems to be an instance of the hardware shipping with buggy firmware which wasn't problematic in some circumstances previously but is with newer drivers (and is probably the same on newer drivers with Windows and Linux too).

marian78

Today I decided to disasemble pfSense box and exchange again RAM modules and Intel network card HP NC360T (in a drawer I found some realtek PCIe 1x).

The 1. problem - network card on the board (NC107i - BCM5723) not working - after a closer look, I found a solution. It was a mistake between my keyboard and my chair. :) In the BIOS I forgot to turn on "Option ROM boot". So sorry :( , that i flame this forum with this problem, it is now functional.

The 2. problem - random reboots caused by intel card NC360T - I replace it with Realtek PCIe 1x card.

After these changes, I did a stress test. I start downloads: 2x PCBSD iso (via http) + 2x openSUSE iso (via http) + 1x openSUSE (via torrent) from FreeNAS server.

Maximum downloads I reached 60-80 Mb / s.

Until now i had no reboot on the pfSense box. ;D

I will continue with testing and i will send the result.

edwardwong

@marian78:

Today I decided to disasemble pfSense box and exchange again RAM modules and Intel network card HP NC360T (in a drawer I found some realtek PCIe 1x).

The 1. problem - network card on the board (NC107i - BCM5723) not working - after a closer look, I found a solution. It was a mistake between my keyboard and my chair. :) In the BIOS I forgot to turn on "Option ROM boot". So sorry :( , that i flame this forum with this problem, it is now functional.

The 2. problem - random reboots caused by intel card NC360T - I replace it with Realtek PCIe 1x card.

After these changes, I did a stress test. I start downloads: 2x PCBSD iso (via http) + 2x openSUSE iso (via http) + 1x openSUSE (via torrent) from FreeNAS server.

Maximum downloads I reached 60-80 Mb / s.

Until now i had no reboot on the pfSense box. ;D

I will continue with testing and i will send the result.

Try to pull a much bigger file, referring to what I mentioned about those tweaks, it works apparently, but when I was pulling something > 5GB continuously problem comes up again.
BTW the download speed is a bit disappointed…..

marian78

Ok, i try download 40G iso.
For now all woking well, without any problem.

Thx all fo yours time.

edwardwong

Nice to see it's working, but personally I don't really like the Realtek card, and the onboard Broadcom doesn't support Jumbo Frame as well….