HP T730 help please

cgi2099

Where is the best place to access the system logs with WinSCP? The ones on the web GUI aren't giving me much

cgi2099

Also should I change:

net.inet.tcp.tso set to 1

to

net.inet.tcp.tso set to 0

cgi2099

@stephenw10

ixl0: <Intel(R) Ethernet Controller X710 for 10GBASE-T - 2.3.1-k> mem 0xe1000000-0xe1ffffff,0xe2008000-0xe200ffff at device 0.0 on pci1
ixl0: fw 7.2.60285 api 1.9 nvm 7.21 etid 80007a1a oem 1.266.0
ixl0: PF-ID[0]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, MDIO shared
ixl0: Using 1024 TX descriptors and 1024 RX descriptors
ixl0: Using 4 RX queues 4 TX queues
ixl0: Using MSI-X interrupts with 5 vectors
ixl0: Ethernet address: 68:05:ca:c1:8b:e0
ixl0: Allocating 4 queues for PF LAN VSI; 4 queues active
ixl0: PCI Express Bus: Speed 8.0GT/s Width x8
ixl0: SR-IOV ready
ixl0: netmap queues/slots: TX 4/1024, RX 4/1024
ixl0: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: No
ixl0: link state changed to UP
ixl0: link state changed to DOWN
ixl0: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: No
ixl0: link state changed to UP
ixl0: promiscuous mode enabled
ixl0: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-0)
ixl0: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-0)

cgi2099

@stephenw10

Here is another:

ixl1: <Intel(R) Ethernet Controller X710 for 10GBASE-T - 2.3.1-k> mem 0xe0000000-0xe0ffffff,0xe2000000-0xe2007fff at device 0.1 on pci1
ixl1: fw 7.2.60285 api 1.9 nvm 7.21 etid 80007a1a oem 1.266.0
ixl1: PF-ID[1]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, MDIO shared
ixl1: Using 1024 TX descriptors and 1024 RX descriptors
ixl1: Using 4 RX queues 4 TX queues
ixl1: Using MSI-X interrupts with 5 vectors
ixl1: Ethernet address: 68:05:ca:c1:8b:e1
ixl1: Allocating 4 queues for PF LAN VSI; 4 queues active
ixl1: PCI Express Bus: Speed 8.0GT/s Width x8
ixl1: SR-IOV ready
ixl1: netmap queues/slots: TX 4/1024, RX 4/1024
ixl1: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: None
ixl1: link state changed to UP
ixl1: link state changed to DOWN
ixl1: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: No
ixl1: link state changed to UP
ixl1: promiscuous mode enabled
ixl1: Malicious Driver Detection event 1 on RX queue 771, pf number 0 (PF-1)
ixl1: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-1)
ixl1: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-1)

cgi2099

@stephenw10

ntopng is causing the spikes I believe, there are about 10 of these that hit at the same time ~8-10% usage on all of them:

/usr/local/bin/ntopng -U ntopng -G /var/run/ntopng/ntopng.pid -1 /usr/local/share/ntopng/httpdocs -2 /usr/local/share/ntopng/scripts -3 /usr/local/share/ntopng/scripts/callbacks -e{ntopng}

cgi2099

@cgi2099

ntopng is definitely the culprit on the spikes, with it disabled never goes above 40% usage even with 5 opvn users.

Is this normal behavior for ntopng?

stephenw10

The system log is in /var/log/system.log

ntopng can use significant CPU. If it's maxing out any cpu core it will cause latency issues.

Steve

cgi2099

@stephenw10

So a few things were happening, I think.

I don't work Fridays and someone decided to plug a cable into the realtek port. I guess this was causing the crashing?

I ended up reinstalling everything and my usage now is always less than 50% and is normally at 2-5%.

Is it normal for the realtek port to cause hard crashes like this without any logs or reports?

But no crashes since, fingers crossed

stephenw10

@cgi2099 said in HP T730 help please:

Is it normal for the realtek port to cause hard crashes like this without any logs or reports?

No. Usually the Realtek hardware/driver stop talking to each other and you see the watchdog timeout errors logged but it doesn't affect any other part of the firewall. Many systems only have Realtek NICs though and in that situation you lose connection. The console should still work though.

Steve

cgi2099

@stephenw10

Well, I spoke too soon; everything went well pretty much all day, ~8 hours. But then it kind of crashed this time. Not like before though.

This time everything lost internet. I could still access the console and web GUI this time. But nothing had internet. Also no errors in the console or log.

I had to reboot to get internet back. Any ideas?

stephenw10

What errors were shown when you tried to access anything?

Did you test from the pfSense command line, trying to ping out for example?

If nothing at all was logged it's hard to say...

cgi2099

@stephenw10

I was able to ping internally but could not ping google or cloudflare.

Nothing logged though, from what I can tell, to me it was like the wan port stopped working.

I will dig another in winscp and see if I can find any more logs somewhere.

If it was having RAM errors or SSD errors would they get logged somewhere?

stephenw10

Bad RAM usually results in random crashes. A bad SSD can result in odd failures where services fail over some hours and logging obviously isn't possible. But you would see errors at the console.

When you tried to ping google.com what error was shown?

cgi2099

@stephenw10

It just timed out when I pinged 8.8.8.8.

It's there a mem test built in?

Logging i believe is working, well it is for ntopng, netdata and adguard.

The only errors I am getting on the console are still:

ixl1: Malicious Driver Detection event 1 on RX queue 771, pf number 0 (PF-1)
ixl1: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-1)
ixl1: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-1)

Same for ixl0

stephenw10

Ah, yes. Hitting that issue will prevent traffic but should not crash the OS which sounds like what you are now seeing?
Previously it looked like the entire firewall was crashing which would not have been that.

cgi2099

@stephenw10

For the above error I read somewhere to try the newer drivers and or increase buffer size. But I am unsure how to do either.

stephenw10

You are running 2.6?

You could go to 22.05 or try a 2.7 snapshot. The drivers are slightly newer there though there are no changes I'm aware of that would affect this.

I'm not aware of any workaround for that currently. Which buffer was referenced?

Steve

cgi2099

@stephenw10

Yes on the 2.6.

I think I might go to 22.05, are there any things I need to be aware of switching from CE to Plus?

I guess the buffer doesn't apply to me since I don't have a bridge:

"Most of the issue reports have been from those running a bridge interface with ixl0 and ixl1. However, there have been multiple reports without using bridges as well.
Increasing the buffer size on the bridge reduced the frequency of the events (went from once a day to taking 5 days before it reoccurred)."

I have ordered an Intel i350 to replace the x710, when the card comes in I'll swap out the RAM as well.

stephenw10

Nope the upgrade to 22.05 should be relatively painless.

Yes, with no bridges defined the bridge buffer does nothing. I'm not sure it actually affects this even when there are bridges in play.

Steve

cgi2099

@stephenw10

Haven't upgraded yet. But maybe have some more useful info.

Today it has been "crashing" like before. But this time the console still works. When it "crashes" now I lose access to the web gui and all internet traffic goes down.

Only errors are the ones from before, nothing else on the console.