HP T730 help please
-
@stephenw10
No bridge that I am aware of unless it automatically did during the install.I am using one of the ports for the wan and the other for the lan. The realtek is not being used.
The console works right up to the point where everything goes down and then everything just has a black screen or loses connection.
The device shows to still be on, though with the activity lights on the nic blinking like there is still data going through.
I will try ctl+t next time this happens.
Everything seems to be working great while its up other than the 100% CPU spikes, I'd say the spike are happening about every 2 minutes or so.
-
Anything in the system logs at around the same 2min interval?
-
Where is the best place to access the system logs with WinSCP? The ones on the web GUI aren't giving me much
-
-
ixl0: <Intel(R) Ethernet Controller X710 for 10GBASE-T - 2.3.1-k> mem 0xe1000000-0xe1ffffff,0xe2008000-0xe200ffff at device 0.0 on pci1 ixl0: fw 7.2.60285 api 1.9 nvm 7.21 etid 80007a1a oem 1.266.0 ixl0: PF-ID[0]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, MDIO shared ixl0: Using 1024 TX descriptors and 1024 RX descriptors ixl0: Using 4 RX queues 4 TX queues ixl0: Using MSI-X interrupts with 5 vectors ixl0: Ethernet address: 68:05:ca:c1:8b:e0 ixl0: Allocating 4 queues for PF LAN VSI; 4 queues active ixl0: PCI Express Bus: Speed 8.0GT/s Width x8 ixl0: SR-IOV ready ixl0: netmap queues/slots: TX 4/1024, RX 4/1024 ixl0: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: No ixl0: link state changed to UP ixl0: link state changed to DOWN ixl0: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: No ixl0: link state changed to UP ixl0: promiscuous mode enabled ixl0: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-0) ixl0: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-0) -
Here is another:
ixl1: <Intel(R) Ethernet Controller X710 for 10GBASE-T - 2.3.1-k> mem 0xe0000000-0xe0ffffff,0xe2000000-0xe2007fff at device 0.1 on pci1 ixl1: fw 7.2.60285 api 1.9 nvm 7.21 etid 80007a1a oem 1.266.0 ixl1: PF-ID[1]: VFs 64, MSI-X 129, VF MSI-X 5, QPs 768, MDIO shared ixl1: Using 1024 TX descriptors and 1024 RX descriptors ixl1: Using 4 RX queues 4 TX queues ixl1: Using MSI-X interrupts with 5 vectors ixl1: Ethernet address: 68:05:ca:c1:8b:e1 ixl1: Allocating 4 queues for PF LAN VSI; 4 queues active ixl1: PCI Express Bus: Speed 8.0GT/s Width x8 ixl1: SR-IOV ready ixl1: netmap queues/slots: TX 4/1024, RX 4/1024 ixl1: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: None ixl1: link state changed to UP ixl1: link state changed to DOWN ixl1: Link is up, 1 Gbps Full Duplex, Requested FEC: None, Negotiated FEC: CL74 FC-FEC/BASE-R, Autoneg: True, Flow Control: No ixl1: link state changed to UP ixl1: promiscuous mode enabled ixl1: Malicious Driver Detection event 1 on RX queue 771, pf number 0 (PF-1) ixl1: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-1) ixl1: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-1) -
ntopng is causing the spikes I believe, there are about 10 of these that hit at the same time ~8-10% usage on all of them:
/usr/local/bin/ntopng -U ntopng -G /var/run/ntopng/ntopng.pid -1 /usr/local/share/ntopng/httpdocs -2 /usr/local/share/ntopng/scripts -3 /usr/local/share/ntopng/scripts/callbacks -e{ntopng} -
ntopng is definitely the culprit on the spikes, with it disabled never goes above 40% usage even with 5 opvn users.
Is this normal behavior for ntopng?
-
The system log is in /var/log/system.log
ntopng can use significant CPU. If it's maxing out any cpu core it will cause latency issues.
Steve
-
So a few things were happening, I think.
I don't work Fridays and someone decided to plug a cable into the realtek port. I guess this was causing the crashing?
I ended up reinstalling everything and my usage now is always less than 50% and is normally at 2-5%.
Is it normal for the realtek port to cause hard crashes like this without any logs or reports?
But no crashes since, fingers crossed
-
@cgi2099 said in HP T730 help please:
Is it normal for the realtek port to cause hard crashes like this without any logs or reports?
No. Usually the Realtek hardware/driver stop talking to each other and you see the watchdog timeout errors logged but it doesn't affect any other part of the firewall. Many systems only have Realtek NICs though and in that situation you lose connection. The console should still work though.
Steve
-
Well, I spoke too soon; everything went well pretty much all day, ~8 hours. But then it kind of crashed this time. Not like before though.
This time everything lost internet. I could still access the console and web GUI this time. But nothing had internet. Also no errors in the console or log.
I had to reboot to get internet back. Any ideas?
-
What errors were shown when you tried to access anything?
Did you test from the pfSense command line, trying to ping out for example?
If nothing at all was logged it's hard to say...
-
I was able to ping internally but could not ping google or cloudflare.
Nothing logged though, from what I can tell, to me it was like the wan port stopped working.
I will dig another in winscp and see if I can find any more logs somewhere.
If it was having RAM errors or SSD errors would they get logged somewhere?
-
Bad RAM usually results in random crashes. A bad SSD can result in odd failures where services fail over some hours and logging obviously isn't possible. But you would see errors at the console.
When you tried to ping google.com what error was shown?
-
It just timed out when I pinged 8.8.8.8.
It's there a mem test built in?
Logging i believe is working, well it is for ntopng, netdata and adguard.
The only errors I am getting on the console are still:
ixl1: Malicious Driver Detection event 1 on RX queue 771, pf number 0 (PF-1) ixl1: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-1) ixl1: Malicious Driver Detection event 1 on RX queue 769, pf number 0 (PF-1)Same for ixl0
-
Ah, yes. Hitting that issue will prevent traffic but should not crash the OS which sounds like what you are now seeing?
Previously it looked like the entire firewall was crashing which would not have been that. -
For the above error I read somewhere to try the newer drivers and or increase buffer size. But I am unsure how to do either.
-
You are running 2.6?
You could go to 22.05 or try a 2.7 snapshot. The drivers are slightly newer there though there are no changes I'm aware of that would affect this.
I'm not aware of any workaround for that currently. Which buffer was referenced?
Steve
-
Yes on the 2.6.
I think I might go to 22.05, are there any things I need to be aware of switching from CE to Plus?
I guess the buffer doesn't apply to me since I don't have a bridge:
"Most of the issue reports have been from those running a bridge interface with ixl0 and ixl1. However, there have been multiple reports without using bridges as well.
Increasing the buffer size on the bridge reduced the frequency of the events (went from once a day to taking 5 days before it reoccurred)."I have ordered an Intel i350 to replace the x710, when the card comes in I'll swap out the RAM as well.
