Very odd behaviour on 2.4.5 vs 2.4.4
-
I did the in place upgrade, and in terms of the pfsense UI itself everything is fine.
However the performance of the firewall has become extremely bad.
I have spent hours trying to figure out whats going on but I think the best way to describe the symptoms are as follows.
Very slow lookups 2-3s per lookup, not on every lookup its random. (same on dns resolver and forwarder and direct lookups)
Also I noticed lots of stalled connections from dev tools in the browser so e.g. on the paypal payment screen which is usually loading all at once, I can visibly see icons needing 1-2s each to appear one at a time.
If I do speedtests its full speed, so its not a throughout issue, but seems to be an issue like the device is struggling to make initial connections on both UDP and TCP.
I went back to 2.4.4 which was a pain, and everything was back to normal, tried the upgrade again and the lag is back.
Any ideas what might be going on here?
is intel cpu and igb network hardware. SSD for storage, and ram disks, graphs look fine no memory exhaustion, no cpu bottleneck.
It is as if I have an artificial 500ms latency or something.
-
I may have already fixed this, I will hold off a day or so to be sure, then I will inform the community what the problem was.
-
Ok so everything appears normal again.
The issue I believe is related to the igb driver.
I had it set to use 2 queues, configured in loader.conf.local, after I reverted it to single queue mode the issues went as fast as they came. I dont know what the default is now days.
Some may remember setting the driver to two queues on some igb hardware used to cause kernel panics, but then that got fixed in FreeBSD, so the dual queue mode does have a history of problems. If anyone else has these symptoms and they using igb, then this might be a place to look.
The variable is 'hw.igb.num_queues'.