HP T730 help please

cgi2099

@stephenw10
No bridge that I am aware of unless it automatically did during the install.

I am using one of the ports for the wan and the other for the lan. The realtek is not being used.

The console works right up to the point where everything goes down and then everything just has a black screen or loses connection.

The device shows to still be on, though with the activity lights on the nic blinking like there is still data going through.

I will try ctl+t next time this happens.

Everything seems to be working great while its up other than the 100% CPU spikes, I'd say the spike are happening about every 2 minutes or so.

stephenw10

Anything in the system logs at around the same 2min interval?

cgi2099

@stephenw10

Where is the best place to access the system logs with WinSCP? The ones on the web GUI aren't giving me much

cgi2099

@stephenw10

Also should I change:

net.inet.tcp.tso set to 1

to

net.inet.tcp.tso set to 0

cgi2099

@stephenw10

ixl0: <Intel(R) Ethernet Controller X710 for 10GBASE-T - 2.3.1-k> mem 0xe1000000-0xe1ffffff,0xe2008000-0xe200ffff at device 0.0 on pci1
ixl0: fw 7.2.60285 api 1.9 nvm 7.21 etid 80007a1a oem 1.266.0
ixl0: PF-ID[0]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, MDIO shared
ixl0: Using 1024 TX descriptors and 1024 RX descriptors
ixl0: Using 4 RX queues 4 TX queues
ixl0: Using MSI-X interrupts with 5 vectors
ixl0: Ethernet address: 68:05:ca:c1:8b:e0
ixl0: Allocating 4 queues for PF LAN VSI; 4 queues active
ixl0: PCI Express Bus: Speed 8.0GT/s Width x8
ixl0: SR-IOV ready
ixl0: netmap queues/slots: TX 4/1024, RX 4/1024
ixl0: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: No
ixl0: link state changed to UP
ixl0: link state changed to DOWN
ixl0: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: No
ixl0: link state changed to UP
ixl0: promiscuous mode enabled
ixl0: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-0)
ixl0: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-0)

cgi2099

@stephenw10

Here is another:

ixl1: <Intel(R) Ethernet Controller X710 for 10GBASE-T - 2.3.1-k> mem 0xe0000000-0xe0ffffff,0xe2000000-0xe2007fff at device 0.1 on pci1
ixl1: fw 7.2.60285 api 1.9 nvm 7.21 etid 80007a1a oem 1.266.0
ixl1: PF-ID[1]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, MDIO shared
ixl1: Using 1024 TX descriptors and 1024 RX descriptors
ixl1: Using 4 RX queues 4 TX queues
ixl1: Using MSI-X interrupts with 5 vectors
ixl1: Ethernet address: 68:05:ca:c1:8b:e1
ixl1: Allocating 4 queues for PF LAN VSI; 4 queues active
ixl1: PCI Express Bus: Speed 8.0GT/s Width x8
ixl1: SR-IOV ready
ixl1: netmap queues/slots: TX 4/1024, RX 4/1024
ixl1: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: None
ixl1: link state changed to UP
ixl1: link state changed to DOWN
ixl1: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: No
ixl1: link state changed to UP
ixl1: promiscuous mode enabled
ixl1: Malicious Driver Detection event 1 on RX queue 771, pf number 0 (PF-1)
ixl1: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-1)
ixl1: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-1)

cgi2099

@stephenw10

ntopng is causing the spikes I believe, there are about 10 of these that hit at the same time ~8-10% usage on all of them:

/usr/local/bin/ntopng -U ntopng -G /var/run/ntopng/ntopng.pid -1 /usr/local/share/ntopng/httpdocs -2 /usr/local/share/ntopng/scripts -3 /usr/local/share/ntopng/scripts/callbacks -e{ntopng}

cgi2099

@cgi2099

ntopng is definitely the culprit on the spikes, with it disabled never goes above 40% usage even with 5 opvn users.

Is this normal behavior for ntopng?

stephenw10

The system log is in /var/log/system.log

ntopng can use significant CPU. If it's maxing out any cpu core it will cause latency issues.

Steve

cgi2099

@stephenw10

So a few things were happening, I think.

I don't work Fridays and someone decided to plug a cable into the realtek port. I guess this was causing the crashing?

I ended up reinstalling everything and my usage now is always less than 50% and is normally at 2-5%.

Is it normal for the realtek port to cause hard crashes like this without any logs or reports?

But no crashes since, fingers crossed

stephenw10

@cgi2099 said in HP T730 help please:

Is it normal for the realtek port to cause hard crashes like this without any logs or reports?

No. Usually the Realtek hardware/driver stop talking to each other and you see the watchdog timeout errors logged but it doesn't affect any other part of the firewall. Many systems only have Realtek NICs though and in that situation you lose connection. The console should still work though.

Steve

cgi2099

@stephenw10

Well, I spoke too soon; everything went well pretty much all day, ~8 hours. But then it kind of crashed this time. Not like before though.

This time everything lost internet. I could still access the console and web GUI this time. But nothing had internet. Also no errors in the console or log.

I had to reboot to get internet back. Any ideas?

stephenw10

What errors were shown when you tried to access anything?

Did you test from the pfSense command line, trying to ping out for example?

If nothing at all was logged it's hard to say...

cgi2099

@stephenw10

I was able to ping internally but could not ping google or cloudflare.

Nothing logged though, from what I can tell, to me it was like the wan port stopped working.

I will dig another in winscp and see if I can find any more logs somewhere.

If it was having RAM errors or SSD errors would they get logged somewhere?

stephenw10

Bad RAM usually results in random crashes. A bad SSD can result in odd failures where services fail over some hours and logging obviously isn't possible. But you would see errors at the console.

When you tried to ping google.com what error was shown?

cgi2099

@stephenw10

It just timed out when I pinged 8.8.8.8.

It's there a mem test built in?

Logging i believe is working, well it is for ntopng, netdata and adguard.

The only errors I am getting on the console are still:

ixl1: Malicious Driver Detection event 1 on RX queue 771, pf number 0 (PF-1)
ixl1: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-1)
ixl1: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-1)

Same for ixl0

stephenw10

Ah, yes. Hitting that issue will prevent traffic but should not crash the OS which sounds like what you are now seeing?
Previously it looked like the entire firewall was crashing which would not have been that.

cgi2099

@stephenw10

For the above error I read somewhere to try the newer drivers and or increase buffer size. But I am unsure how to do either.

stephenw10

You are running 2.6?

You could go to 22.05 or try a 2.7 snapshot. The drivers are slightly newer there though there are no changes I'm aware of that would affect this.

I'm not aware of any workaround for that currently. Which buffer was referenced?

Steve

cgi2099

@stephenw10

Yes on the 2.6.

I think I might go to 22.05, are there any things I need to be aware of switching from CE to Plus?

I guess the buffer doesn't apply to me since I don't have a bridge:

"Most of the issue reports have been from those running a bridge interface with ixl0 and ixl1. However, there have been multiple reports without using bridges as well.
Increasing the buffer size on the bridge reduced the frequency of the events (went from once a day to taking 5 days before it reoccurred)."

I have ordered an Intel i350 to replace the x710, when the card comes in I'll swap out the RAM as well.