HP T730 help please

cgi2099

@stephenw10
No crash report or logs that I can find.

When it crashes the web gui stops working and I can't ssh in anymore.

Temperatures never get over 40c and ram usage is never above 10%.

stephenw10

Have you tried connecting to the console when that happens?

Or logging the console output would be better if you can. There are some errors that only appear on the console.

cgi2099

@stephenw10

I have not but will try

Do I need a serial cable or just need to plug a monitor in?

stephenw10

Could be either (or both!). It depends how it was installed. If it has a serial port and you enable the serial console you can log it's output on a terminal client which is useful.
It's probably using the video console though so just connect a monitor/keyboard and see if it's still responding there.

Steve

cgi2099

@stephenw10
Nothing was responding serial cable or video. I have been trying all day to get errors and I finally was able to from the console before the screen just went black. Look like there could be more but I can't make them out before everything goes black. Not much but maybe can help:

ixl1: Malicious Driver Detection event 1 on RX queue 771, pf number 0 (PF-1)

stephenw10

Hmm, do you have a bridge configured?

That looks like this known issue: https://redmine.pfsense.org/issues/13003
Though that doesn't normally crash the OS entirely AFAIK.

You should be able to use the console when it running normally, that is functional I assume?

If it just stops responding when this happens try pressing ctl+t that can respond when nothing else will. If it does it should show you what it's waiting for.

Steve

cgi2099

@stephenw10
No bridge that I am aware of unless it automatically did during the install.

I am using one of the ports for the wan and the other for the lan. The realtek is not being used.

The console works right up to the point where everything goes down and then everything just has a black screen or loses connection.

The device shows to still be on, though with the activity lights on the nic blinking like there is still data going through.

I will try ctl+t next time this happens.

Everything seems to be working great while its up other than the 100% CPU spikes, I'd say the spike are happening about every 2 minutes or so.

stephenw10

Anything in the system logs at around the same 2min interval?

cgi2099

@stephenw10

Where is the best place to access the system logs with WinSCP? The ones on the web GUI aren't giving me much

cgi2099

@stephenw10

Also should I change:

net.inet.tcp.tso set to 1

to

net.inet.tcp.tso set to 0

cgi2099

@stephenw10

ixl0: <Intel(R) Ethernet Controller X710 for 10GBASE-T - 2.3.1-k> mem 0xe1000000-0xe1ffffff,0xe2008000-0xe200ffff at device 0.0 on pci1
ixl0: fw 7.2.60285 api 1.9 nvm 7.21 etid 80007a1a oem 1.266.0
ixl0: PF-ID[0]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, MDIO shared
ixl0: Using 1024 TX descriptors and 1024 RX descriptors
ixl0: Using 4 RX queues 4 TX queues
ixl0: Using MSI-X interrupts with 5 vectors
ixl0: Ethernet address: 68:05:ca:c1:8b:e0
ixl0: Allocating 4 queues for PF LAN VSI; 4 queues active
ixl0: PCI Express Bus: Speed 8.0GT/s Width x8
ixl0: SR-IOV ready
ixl0: netmap queues/slots: TX 4/1024, RX 4/1024
ixl0: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: No
ixl0: link state changed to UP
ixl0: link state changed to DOWN
ixl0: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: No
ixl0: link state changed to UP
ixl0: promiscuous mode enabled
ixl0: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-0)
ixl0: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-0)

cgi2099

@stephenw10

Here is another:

ixl1: <Intel(R) Ethernet Controller X710 for 10GBASE-T - 2.3.1-k> mem 0xe0000000-0xe0ffffff,0xe2000000-0xe2007fff at device 0.1 on pci1
ixl1: fw 7.2.60285 api 1.9 nvm 7.21 etid 80007a1a oem 1.266.0
ixl1: PF-ID[1]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, MDIO shared
ixl1: Using 1024 TX descriptors and 1024 RX descriptors
ixl1: Using 4 RX queues 4 TX queues
ixl1: Using MSI-X interrupts with 5 vectors
ixl1: Ethernet address: 68:05:ca:c1:8b:e1
ixl1: Allocating 4 queues for PF LAN VSI; 4 queues active
ixl1: PCI Express Bus: Speed 8.0GT/s Width x8
ixl1: SR-IOV ready
ixl1: netmap queues/slots: TX 4/1024, RX 4/1024
ixl1: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: None
ixl1: link state changed to UP
ixl1: link state changed to DOWN
ixl1: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: No
ixl1: link state changed to UP
ixl1: promiscuous mode enabled
ixl1: Malicious Driver Detection event 1 on RX queue 771, pf number 0 (PF-1)
ixl1: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-1)
ixl1: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-1)

cgi2099

@stephenw10

ntopng is causing the spikes I believe, there are about 10 of these that hit at the same time ~8-10% usage on all of them:

/usr/local/bin/ntopng -U ntopng -G /var/run/ntopng/ntopng.pid -1 /usr/local/share/ntopng/httpdocs -2 /usr/local/share/ntopng/scripts -3 /usr/local/share/ntopng/scripts/callbacks -e{ntopng}

cgi2099

@cgi2099

ntopng is definitely the culprit on the spikes, with it disabled never goes above 40% usage even with 5 opvn users.

Is this normal behavior for ntopng?

stephenw10

The system log is in /var/log/system.log

ntopng can use significant CPU. If it's maxing out any cpu core it will cause latency issues.

Steve

cgi2099

@stephenw10

So a few things were happening, I think.

I don't work Fridays and someone decided to plug a cable into the realtek port. I guess this was causing the crashing?

I ended up reinstalling everything and my usage now is always less than 50% and is normally at 2-5%.

Is it normal for the realtek port to cause hard crashes like this without any logs or reports?

But no crashes since, fingers crossed

stephenw10

@cgi2099 said in HP T730 help please:

Is it normal for the realtek port to cause hard crashes like this without any logs or reports?

No. Usually the Realtek hardware/driver stop talking to each other and you see the watchdog timeout errors logged but it doesn't affect any other part of the firewall. Many systems only have Realtek NICs though and in that situation you lose connection. The console should still work though.

Steve

cgi2099

@stephenw10

Well, I spoke too soon; everything went well pretty much all day, ~8 hours. But then it kind of crashed this time. Not like before though.

This time everything lost internet. I could still access the console and web GUI this time. But nothing had internet. Also no errors in the console or log.

I had to reboot to get internet back. Any ideas?

stephenw10

What errors were shown when you tried to access anything?

Did you test from the pfSense command line, trying to ping out for example?

If nothing at all was logged it's hard to say...

cgi2099

@stephenw10

I was able to ping internally but could not ping google or cloudflare.

Nothing logged though, from what I can tell, to me it was like the wan port stopped working.

I will dig another in winscp and see if I can find any more logs somewhere.

If it was having RAM errors or SSD errors would they get logged somewhere?