Upload Speed issue and WebUI intermittent hangs
I'm investigating a curious case of intermittent latency and WebUI hangs on specific pages.
The setup is as follows:
Fiber to Ethernet converter (ISP provided, known good) -> pppoe0 (via em4)
ATM to Ethernet converter (ADSL 2+ Modem) -> pppoe1 (via igb0)
em0+em1+em2+em3 -> lagg0 (with a few VLANS)
few VLANS in a lagg0 -> HP managed switch -> lan clients
So basically: 2x WAN and a 4x1Gbit LAGG with a few VLANs exposed via a switch.
Hardware: Supermicro X10SLV-Q with Xeon E3-1225v3 and 16GB RAM, 120GB SSD (Crucial), and an intel quad port nic.
I suspect that somehow the two issues are connected, but I can't be sure, so that's why they are stuffed together into one topic.
The issue is as follows: the fiber connection is a 20/20 capped line and when directly connected does put out a real world http up and down of about 19/19, so that's pretty much on the spot. When pfSense is in between, the download speed is still 20Mbit, but the upload varies for unknown reasons. The tests are done from the firewall itself to exclude the switch, the LAGG and NAT. As a double measure we also unplugged the DSL 2nd WAN and did the same test twice on pfSense and two times via a modem 4G directly attached to a computer to make sure its not the server on the other end. In most cases we get 1Mbit upload, sometimes it suddenly jumps to 10Mbit but then just goes back to 1Mbit or lower. The connection doesn't drop, so it's not like some averaging is happening. The bandwidth graphs are pretty stable too on the WebUI, no zig-zag, just stable 1Mbit upload or stable 10Mbit upload. Nothing in the logs, tcpdump doesn't show drops, drop counters on the interfaces don't increase, it just for some reason fails to push the data out it seems.
Another issue is that editing firewall rules (submitting the form for a new rule or a modified rule) takes over 50 seconds. Applying the rules once the page comes back is instant. Same goes for editing DHCP assignments. Page and form load instantly, but clicking save after an edit or adding an entry takes almost a minute to complete. Applying the settings is instantly done again. It seems to be something related to the method used in PHP to write the config?
I can't find any problems in the logs, the box is 99% idle most of the time in terms of CPU utilisation, all limiters are removed just to check, as well as Suricata. I've also downloaded the config.xml and tried to find any problems in there, but it seems that all settings are there, and correct, and the XML validates just fine. Restoring it to a fresh install made no change. The only thing I haven't tried yet is doing a fresh install and manually setting up the WAN PPPoE connection again and trying to see if that makes for a constant upload speed. I might do that this evening just to test it.
I'm wondering if anyone knows any other debugging or diagnostic methods that I haven't tried yet, short of completely replacing the box?
The ACB server has been having issues for many months now. Getting really annoying.