Issue with pfsense taking a nose dive until reboot
-
So I have setup a few of these pfsense routers with no issue.
I set one up at my house some time ago to vpn into my office and use snort to scan packets going in and out.I have never fixed this problem but its annoying me enough to at least finally ask.
I assume this is a hardware issue somehow, because I have never encountered it and do not have this issue on any other pfsense system I setup. I could be wrong though.TLDR:
It seems this is problem is about total bandwidth going through the pfsense router, and not individual devices.If I download more than 100mbps or so at a sustained speed for more than 5 minutes or so, my pfsense takes a nose dive and becomes nearly unresponsive until restart, in which this whole situation is resolved. This situation will not resolve itself over time, if i wait an hour it will still be broken until I reboot.
What I know is that the 100mbps or less territory allows sustained speeds with no issues.
Anything above that for more than a hand full of minutes results in a nose dive, the only solution is to hit the power button on the pfsense computer, wait for it to go down and then boot it back up.Currently:
I get maximum speed of my isp from my pfsense router, my ISP provides 200mbps and that is what I receive on speed tests.
The issue:
I download a large file, say 50 GB.
The router will handle this at max speed for a while, say 10~30 GB of the download at full on 200mbps.after some time of this extended speed, the router takes a nose dive.
Speeds reduce dramatically to almost 0 if not 0.
The entire network in my house is in disarray, Chromecast even die out, it seems routing fails. Wifi ethernet, all of it is basically dead or running very slow. Generally wifi devices are basically not running at all, and ethernet devices might somewhat run they probably will not go on a website, but might be able to load the configuration menu for pfsense, but very slow, switching between pages is painful if it works at all.
The network stays this way until I reboot pfsense in which case everything is now back to normal as if nothing happened at all.My temporary solution:
I have limited all my download speeds on things like steam to less then 10MBps or roughly 100mbps , I can download at this speed indefinitely as long as nothing else is doing something similar, this means if I have two computers downloading at 100mbps, I can expect this nose dive to take place.It seems this is about total bandwidth going through the pfsense router, and not individual devices.
It doesn't matter if I am going through WIFI (Ubiquiti Unifi AP's) or ethernet, this will happen either way if my sustained total bandwidth goes too high.
I am not totally sure about the ceiling of this bandwidth, if its 100mbps sustained, or 120mbps etc.
What I know is that the 100mbps or less territory allows sustained speeds with no issues.
And if the nose dive takes place, the only solution is to hit the power button on the pfsense computer, wait for it to go down and then boot it back up. This situation will not resolve itself over time, if I wait an hour it will still be broken until I reboot.Any thoughts?
-
@rigidconduit said in Issue with pfsense taking a nose dive until reboot:
Any thoughts?
Yeah.
A simple question first : what do you think what snort 'sees' ?
If your traffic is like the average 'Internet' traffic, then you have no
plain emails
http
etc.
Its all TLS these days.
snort can't do nothing with TLS, as it is just seen as 'pure random bianry'. All it knows, is source address and port, and destination address and port.edit : maybe some DNS packets with rather harmful info can be 'analysed' by snort ^^
Next : do this test : http://www.dslreports.com/speedtest
Depending on the type of connection used, a big download can block upload 'control' packets.
Just to motivate you : never saw what you described.
Also, go console, option 8 (and pkg install htop)
Use top or htop during the download.
What process is taking the most resources ? -
Snort probably doesn't do a whole lot, I really do not see much activity other than dshield blocks and the occasional sipvicous scan etc (not that I am running a phone server), at the office it could be a bit more useful. Here at the house its enabled because buttons existed :D lol probably not the best answer but I did it for the sake of adding it (which is not always best practice).
My main purpose is to link my house to the office to make things easy for me at home when it comes to working.
Anyhow I dont think overall it has an effect on the current issue tbh??(could be wrong)
But I dont remember if I have ever tried crashing the pfsense router with it off, if so that would of been the first few months I was using it to try and troubleshoot what the cause is.But that aside
I can tell you that with the system at full chat (200mbps) and snort on I see maybe 10% cpu usage and 13% ram usage (8GB) and the cpu temps never go over 50c.Here is the report output
http://www.dslreports.com/speedtest/67867354I will give htop a whirl next post
-
What network cards are in it?
You can pretty quickly stop Snort and see if that has any effect but I would expect not.@rigidconduit said in Issue with pfsense taking a nose dive until reboot:
dshield blocks
FYI pfBlockerNG's ISC_Block list is DShield, or the ET_Block list includes DShield, if you want to disable those rules in Snort and put them into a firewall rule.
-
Snort Tops the leader board followed shortly by kernel when running htop
With snort disabled its just kernel followed by php-fpmI am trying to crash it now with snort off to see what happens, I am also going to try and disable a few other services that might not be crucial to see the effect.
-
@steveits said in Issue with pfsense taking a nose dive until reboot:
What network cards are in it?
I am not sure if there is a way to identify the cards through pfsense, I have never tried. I am sort of guessing at the integrated..There are 3 network cards in it, the integrated, and 2 third party cards
The two third party cards are these
Rosewill RNG-407-Dualv2
https://www.newegg.com/rosewill-rng-407-dualv2/p/N82E16833166096
The integrated is:
Realtek RTL8111GN - I believe if i pulled the right spec sheet I will need to confirm this though.The third party nics are not totally in use, only one is used and with a single port to run the lan and wifi.
I had a much more complicated setup to dedicate a network to my wife's work computer but have since torn this down for the sake of simplicity.The integrated nic has the wan attached to it.
The computer is either an m90p m92p or e73 thinkcentre, my money is on it being an m92p.
I should add, this has been the issue since day one of using this computer as a pfsense router. It has been like 2 years of this :(
-
The Interfaces/Assignments page will show the interface names, which is based off the driver being used (re0 etc.). I have seen many people complain in this forum about Realtek drivers in FreeBSD, though in the one non-Netgate hardware device I've managed it's not been a problem. Since the WAN interface is using Realtek then you could try moving WAN to the unused interface and see if the problem continues.
-
@steveits
If that is the case, what is em ? o.0I must be wrong about the spec sheet I pulled then which I guess is a little expected, I use 3 different computers at work and they all look almost completely identical, but hardware is different, I must of got the model wrong. This particular machine is just a spare machine off the floor that I grabbed.
It looks like the 4 nics provided by the third party cards are realtek then.
I am not sure the effects of these having issues but IF its only the card having trouble and nothing else, it will explain why I lose the lan as well then.
In which case if I swap the two cables I (in thoery?) lose the internet but not the lan.In the case of the picture, the 'WIFI' interface is the LAN just relabeled
-
@rigidconduit said in Issue with pfsense taking a nose dive until reboot:
what is em
Intel so that's very likely not the problem.
-
@steveits
Going to attempt replacing the nic when i have some time, I will update here with results.I am concluding its most likely an issue with the nic. as far as I have seen it seems to match up with symptoms.
I will leave this post open till I do so and post the results for future reference of anyone that may have this same problem.
-
@rigidconduit said in Issue with pfsense taking a nose dive until reboot:
http://www.dslreports.com/speedtest/67867354
To get full A's, see this forum biggest thread here Home pfSense
Software Traffic ShapingStay away from 're' NIC drivers
