Intermittent slowing internet speed on pfsense 2.4.5

stephenw10

That sort of throttling I would look for something low level like a speed/duplex mismatch on of the interfaces. Especially since you are only seeing it down. Check the Status > Interfaces page for the link speed and any errors or collisions.

Steve

alitech

Thank you for responding. There are no errors on the interfaces page but Here are some screenshots of the drop outs that I see on the dashboard. Where you see the dip is where everything slows down to a crawl. There are 4 images to show this at different times. This happens 5 or 6 times every hour and I just cant figure this out. I have changed the ethernet cables as well.

pfsense connection drops.png
pfsense connection drops 2.png
pfsense connection drops 3.png
pfsense connection drops 4.png

And no errors shown here

stephenw10

Anything shown in the system log when that happens?

So it's not throttled like that all the time? Is there anything specific you have able to use the trigger it?

Steve

provels

DDOS? Check FW logs?

stephenw10

Could be something like that. Though I would expect to see a lot more blocked packets on WAN if it was.

alitech

Here are some of the system logs. What could be the cause of this?

There are more similar statuses. I dont have openvpn configured, it was installed with pfsense yet it shows it there.

Apr 24 22:28:51	rc.gateway_alarm	3425	>>> Gateway alarm: WAN_DHCP (Addr:**** Alarm:0 RTT:163.560ms RTTsd:219.419ms Loss:15%)
Apr 24 22:28:51	check_reload_status		updating dyndns WAN_DHCP
Apr 24 22:28:51	check_reload_status		Restarting ipsec tunnels
Apr 24 22:28:51	check_reload_status		Restarting OpenVPN tunnels/interfaces
Apr 24 22:28:51	check_reload_status		Reloading filter
Apr 24 22:28:52	php-fpm	34665	/rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Apr 24 22:28:52	php-fpm	34665	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Apr 24 22:30:51	nginx		2020/04/24 22:30:51 [crit] 62965#100454: *126082 SSL_write() failed (13: Permission denied) while processing HTTP/2 connection, client: 192.168.1.2, server: 0.0.0.0:443
Apr 24 22:35:25	rc.gateway_alarm	90246	>>> Gateway alarm: WAN_DHCP (Addr:80.194.29.1 Alarm:1 RTT:243.584ms RTTsd:223.339ms Loss:21%)
Apr 24 22:35:25	check_reload_status		updating dyndns WAN_DHCP
Apr 24 22:35:25	check_reload_status		Restarting ipsec tunnels
Apr 24 22:35:25	check_reload_status		Restarting OpenVPN tunnels/interfaces
Apr 24 22:35:25	check_reload_status		Reloading filter
Apr 24 22:35:26	php-fpm	34665	/rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Apr 24 22:35:26	php-fpm	34665	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Apr 24 22:35:57	rc.gateway_alarm	59789	>>> Gateway alarm: WAN_DHCP (Addr:**** Alarm:0 RTT:277.297ms RTTsd:205.898ms Loss:15%)
Apr 24 22:35:57	check_reload_status		updating dyndns WAN_DHCP
Apr 24 22:35:57	check_reload_status		Restarting ipsec tunnels
Apr 24 22:35:57	check_reload_status		Restarting OpenVPN tunnels/interfaces
Apr 24 22:35:57	check_reload_status		Reloading filter
Apr 24 22:35:58	php-fpm	61711	/rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Apr 24 22:35:58	php-fpm	61711	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Apr 24 22:38:25	nginx		2020/04/24 22:38:25 [crit] 62883#100411: *126174 SSL_write() failed (13: Permission denied) while processing HTTP/2 connection, client: 192.168.1.2, server: 0.0.0.0:443
Apr 24 22:38:36	php-fpm	34665	/index.php: Successful login for user 'admin' from: 192.168.1.56 (Local Database)

alitech

Just happened again

I know its not the internet at my end, it works fine with a vigor router

Apr 24 22:45:45	rc.gateway_alarm	53475	>>> Gateway alarm: WAN_DHCP (Addr:**** Alarm:1 RTT:102.902ms RTTsd:179.060ms Loss:21%)
Apr 24 22:45:45	check_reload_status		updating dyndns WAN_DHCP
Apr 24 22:45:45	check_reload_status		Restarting ipsec tunnels
Apr 24 22:45:45	check_reload_status		Restarting OpenVPN tunnels/interfaces
Apr 24 22:45:45	check_reload_status		Reloading filter
Apr 24 22:45:46	php-fpm	34665	/rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Apr 24 22:45:46	php-fpm	34665	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Apr 24 22:46:39	rc.gateway_alarm	15150	>>> Gateway alarm: WAN_DHCP (Addr:**** Alarm:0 RTT:218.963ms RTTsd:221.156ms Loss:13%)
Apr 24 22:46:39	check_reload_status		updating dyndns WAN_DHCP
Apr 24 22:46:39	check_reload_status		Restarting ipsec tunnels
Apr 24 22:46:39	check_reload_status		Restarting OpenVPN tunnels/interfaces
Apr 24 22:46:39	check_reload_status		Reloading filter
Apr 24 22:46:40	php-fpm	34665	/rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
Apr 24 22:46:40	php-fpm	34665	/rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'

stephenw10

Well it's showing some very bad gateway latency and/or packet loss that is reloading stuff.

That could be the gatewau itself or you could be hitting this: https://redmine.pfsense.org/issues/10414
Check the System Activity output when you are seeing this issue if you can.

If you are confident the gateway is not actually having issues you could set the monitoring IP to something else like 8.8.8.8.
You can also disable the monitoring action do it does not reload the firewall rules even if if it sees high pings.
https://docs.netgate.com/pfsense/en/latest/routing/gateway-settings.html

Steve

alitech

Thank you Steve.

I have done what you have recommended screencapture-192-168-1-1-system-gateways-edit-php-2020-04-25-14_05_22.png .

I will provide logs if anything shows up in the system logs.

Thanks a bunch

Cool_Corona

Remove Block Bogons on the interfaces. Then the filter reload doesnt take so much power....

Cool_Corona

@stephenw10

Same output in the logs on all boxes when filter reloads.

alitech

Steve, it seems that the monitoring was causing the issue. I have set it up so it always assumes that the connection is live. Also nothing is reloaded as a consequence. So far the system is behaving, however I have experienced a slight lag in the connection for a couple of times 1 or 2 mins each time.

I am still monitoring the situation. Hopefully it behaves. I will of course report any abnormalities.

@Cool_Corona I have removed the block on bogon networks as well on the WAN.

stephenw10

The gateway monitoring action was likely exposing the problem but should not be an issue in itself.
It may have been triggering too frequently if an external target was not set. ISP gateways are not optimised to reply to ping, the opposite is sometimes true. But even so a gateway event should not be that disruptive/expensive. It could well have been hitting this too: https://redmine.pfsense.org/issues/10414
We are actively working to resolve that.

Anyway glad you're up and running.

Steve

alitech

Thank you Steve, against that bug, I have also reduced the firewall maximum entries to 65534. Bogon is also disabled.

Might be the case with my ISP, I will ask in the dedicated ISP forums for advice on monitoring. There are a lof of pfsense users with Virgin Media in the UK. Helps to drop the ISP name in this thread as well, in case anyone else is going through the same pain.