Intermittent slowing internet speed on pfsense 2.4.5

[alitech]

Hi guys

I set this up about 3 weeks ago and I am not giving up on pfsense yet. I have been using an netgear orbi before with little to know bottlenecks or speed issues but since pfsense, i have had a lot of slow internet speeds which get resolved after a reboot but then goes slow again. This is mostly on either resolving new DNS queries or downloading pics and images on sites.

Youtube videos are sluggish too and are not running full 4k anymore unless forced, which buffers for a while.

I was running snort for a while which pushed up Memory usage way over 80% despite having 8GB on my rig. I have since removed the snort package. I do have suricata running which uses about 10% of RAM and the performance of the internet is the same despite suricata on or not so I know that suricata is not the bottleneck.

After some googling I have enabled DNS forwarding mode, but thats the only other thing I have done. Apart from that everything is vanilla.

I have a 350MB connection with 35mb uplink. I get about 7MB down and 35MB up. This is not acceptable. It reverts to full speed if pfsense is removed from the equation though.

[stephenw10]

That sort of throttling I would look for something low level like a speed/duplex mismatch on of the interfaces. Especially since you are only seeing it down. Check the Status > Interfaces page for the link speed and any errors or collisions.

Steve

[alitech]

Thank you for responding. There are no errors on the interfaces page but Here are some screenshots of the drop outs that I see on the dashboard. Where you see the dip is where everything slows down to a crawl. There are 4 images to show this at different times. This happens 5 or 6 times every hour and I just cant figure this out. I have changed the ethernet cables as well.

And no errors shown here

[stephenw10]

Anything shown in the system log when that happens?

So it's not throttled like that all the time? Is there anything specific you have able to use the trigger it?

Steve

[provels]

DDOS? Check FW logs?

[stephenw10]

Could be something like that. Though I would expect to see a lot more blocked packets on WAN if it was.

[alitech]

Here are some of the system logs. What could be the cause of this?

There are more similar statuses. I dont have openvpn configured, it was installed with pfsense yet it shows it there.

Apr 24 22:28:51	rc.gateway_alarm	3425	>>> Gateway alarm: WAN_DHCP (Addr:**** Alarm:0 RTT:163.560ms RTTsd:219.419ms Loss:15%)
Apr 24 22:28:51	check_reload_status		updating dyndns WAN_DHCP
Apr 24 22:28:51	check_reload_status		Restarting ipsec tunnels
Apr 24 22:28:51	check_reload_status		Restarting OpenVPN tunnels/interfaces
Apr 24 22:28:51	check_reload_status		Reloading filter
Apr 24 22:28:52	php-fpm	34665	/rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Apr 24 22:28:52	php-fpm	34665	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Apr 24 22:30:51	nginx		2020/04/24 22:30:51 [crit] 62965#100454: *126082 SSL_write() failed (13: Permission denied) while processing HTTP/2 connection, client: 192.168.1.2, server: 0.0.0.0:443
Apr 24 22:35:25	rc.gateway_alarm	90246	>>> Gateway alarm: WAN_DHCP (Addr:80.194.29.1 Alarm:1 RTT:243.584ms RTTsd:223.339ms Loss:21%)
Apr 24 22:35:25	check_reload_status		updating dyndns WAN_DHCP
Apr 24 22:35:25	check_reload_status		Restarting ipsec tunnels
Apr 24 22:35:25	check_reload_status		Restarting OpenVPN tunnels/interfaces
Apr 24 22:35:25	check_reload_status		Reloading filter
Apr 24 22:35:26	php-fpm	34665	/rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Apr 24 22:35:26	php-fpm	34665	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Apr 24 22:35:57	rc.gateway_alarm	59789	>>> Gateway alarm: WAN_DHCP (Addr:**** Alarm:0 RTT:277.297ms RTTsd:205.898ms Loss:15%)
Apr 24 22:35:57	check_reload_status		updating dyndns WAN_DHCP
Apr 24 22:35:57	check_reload_status		Restarting ipsec tunnels
Apr 24 22:35:57	check_reload_status		Restarting OpenVPN tunnels/interfaces
Apr 24 22:35:57	check_reload_status		Reloading filter
Apr 24 22:35:58	php-fpm	61711	/rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Apr 24 22:35:58	php-fpm	61711	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Apr 24 22:38:25	nginx		2020/04/24 22:38:25 [crit] 62883#100411: *126174 SSL_write() failed (13: Permission denied) while processing HTTP/2 connection, client: 192.168.1.2, server: 0.0.0.0:443
Apr 24 22:38:36	php-fpm	34665	/index.php: Successful login for user 'admin' from: 192.168.1.56 (Local Database)

[alitech]

Just happened again

I know its not the internet at my end, it works fine with a vigor router

Apr 24 22:45:45	rc.gateway_alarm	53475	>>> Gateway alarm: WAN_DHCP (Addr:**** Alarm:1 RTT:102.902ms RTTsd:179.060ms Loss:21%)
Apr 24 22:45:45	check_reload_status		updating dyndns WAN_DHCP
Apr 24 22:45:45	check_reload_status		Restarting ipsec tunnels
Apr 24 22:45:45	check_reload_status		Restarting OpenVPN tunnels/interfaces
Apr 24 22:45:45	check_reload_status		Reloading filter
Apr 24 22:45:46	php-fpm	34665	/rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Apr 24 22:45:46	php-fpm	34665	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Apr 24 22:46:39	rc.gateway_alarm	15150	>>> Gateway alarm: WAN_DHCP (Addr:**** Alarm:0 RTT:218.963ms RTTsd:221.156ms Loss:13%)
Apr 24 22:46:39	check_reload_status		updating dyndns WAN_DHCP
Apr 24 22:46:39	check_reload_status		Restarting ipsec tunnels
Apr 24 22:46:39	check_reload_status		Restarting OpenVPN tunnels/interfaces
Apr 24 22:46:39	check_reload_status		Reloading filter
Apr 24 22:46:40	php-fpm	34665	/rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Apr 24 22:46:40	php-fpm	34665	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'

[stephenw10]

Well it's showing some very bad gateway latency and/or packet loss that is reloading stuff.

That could be the gatewau itself or you could be hitting this: https://redmine.pfsense.org/issues/10414
Check the System Activity output when you are seeing this issue if you can.

If you are confident the gateway is not actually having issues you could set the monitoring IP to something else like 8.8.8.8.
You can also disable the monitoring action do it does not reload the firewall rules even if if it sees high pings.
https://docs.netgate.com/pfsense/en/latest/routing/gateway-settings.html

Steve

[alitech]

Thank you Steve.

I have done what you have recommended .

I will provide logs if anything shows up in the system logs.

Thanks a bunch

[Cool_Corona]

Remove Block Bogons on the interfaces. Then the filter reload doesnt take so much power....

[Cool_Corona]

@stephenw10

Same output in the logs on all boxes when filter reloads.

[alitech]

Steve, it seems that the monitoring was causing the issue. I have set it up so it always assumes that the connection is live. Also nothing is reloaded as a consequence. So far the system is behaving, however I have experienced a slight lag in the connection for a couple of times 1 or 2 mins each time.

I am still monitoring the situation. Hopefully it behaves. I will of course report any abnormalities.

@Cool_Corona I have removed the block on bogon networks as well on the WAN.

[stephenw10]

The gateway monitoring action was likely exposing the problem but should not be an issue in itself.
It may have been triggering too frequently if an external target was not set. ISP gateways are not optimised to reply to ping, the opposite is sometimes true. But even so a gateway event should not be that disruptive/expensive. It could well have been hitting this too: https://redmine.pfsense.org/issues/10414
We are actively working to resolve that.

Anyway glad you're up and running.

Steve

[alitech]

Thank you Steve, against that bug, I have also reduced the firewall maximum entries to 65534. Bogon is also disabled.

Might be the case with my ISP, I will ask in the dedicated ISP forums for advice on monitoring. There are a lof of pfsense users with Virgin Media in the UK. Helps to drop the ISP name in this thread as well, in case anyone else is going through the same pain.