SG-3100 Packet Loss on 21.02

FeatherKing

I saw it mentioned in a few other threads, but I couldnt see that a new forum post was actually created. I have experienced some strange packet loss on 21.02. It isnt currently happening, but I can could on one hand how many times I have had issues with my ISP and packet loss.

I am aware of the 21.02 lockup issue (which I have also experienced a few times). (perhaps the packet loss isnt happening because I have to reboot the firewall every few hours now)

Maybe the packet loss is nothing, but I figured I would start a thread to see if others are experiencing it as well.

songtancurly

@featherking Did you experience a loss of checking for updates too?
I was running fine using pfBlocker and Snort; after I updated to 21.02, I noticed that snort disappeared and the other apps.
A little later, I noticed that my FW was sluggish, and hard to connect too even on the serial port. My pFBlocker stopped working, I had high CPU utilization and spotty connectivity, and could no longer check for updates.
Long story short, I found that the 21.02 firmware had some issues. I ended up rolling back to 2.4.5p1, which fixed my issues.

In case you don't have the line, here it is talking about the problem:
https://forum.netgate.com/topic/161098/pfsense-plus-and-sg-3100

I think that your issue is another symptom too.

Good luck and take care,

Regards,

William

stephenw10

Packet loss is not a symptom of that issue. If you're hitting that you get a complete loss of all traffic though the device as pf tries to load the ruleset.

What level of packet loss are you seeing?

Steve

FeatherKing

@stephenw10 I dont believe it to be the same issue as complete unavailability across WAN/LAN (i have experience that issue twice and the firewall is totally unavailable even on the LAN interface). And again, this packet loss might be nothing. I have seen a few posts in the forum of people mentioning packet loss since the update. My ISP rarely has problems of this nature, and no outages were reported since friday (that i can see).

For me, I have bounced between 8% and 33% loss at the worst on the WAN interface. At the moment, things seem fine. During the packet loss my CPU was spiking erratically.

Excerpt from the logs during this time

Feb 20 00:07:32 pfSense rc.gateway_alarm[39328]: >>> Gateway alarm: WAN_DHCP (Addr:192.168.100.1 Alarm:1 RTT:2.159ms RTTsd:1.364ms Loss:25%)
Feb 20 00:07:32 pfSense check_reload_status[390]: updating dyndns WAN_DHCP
Feb 20 00:07:32 pfSense check_reload_status[390]: Restarting ipsec tunnels
Feb 20 00:07:32 pfSense check_reload_status[390]: Restarting OpenVPN tunnels/interfaces
Feb 20 00:07:32 pfSense check_reload_status[390]: Reloading filter
Feb 20 00:07:34 pfSense php-fpm[76027]: /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. q'WAN_DHCP'
Feb 20 00:07:34 pfSense php-fpm[76027]: /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. 'WAN_DHCP6'
Feb 20 00:07:35 pfSense php-fpm[88785]: /rc.dyndns.update: Dynamic DNS (zzzz.publicvm.com) There was an error trying to determine the public IP for interface - wan (mvneta2 ).
Feb 20 00:07:42 pfSense rc.gateway_alarm[95940]: >>> Gateway alarm: WAN_DHCP (Addr:192.168.100.1 Alarm:0 RTT:18.314ms RTTsd:57.150ms Loss:8%)

stephenw10

Hmm, looks like that's probably an alarm event and a clear event, check the gateway logs.
10s apart. What was happening just before that triggered? Potentially something putting a huge load on the firewall?

Steve

FeatherKing

@stephenw10 from the gateway logs around that event

Feb 20 00:07:32	dpinger	9777	WAN_DHCP 192.168.100.1: Alarm latency 2159us stddev 1364us loss 25%
Feb 20 00:07:42	dpinger	9777	WAN_DHCP 192.168.100.1: Clear latency 18314us stddev 57150us loss 8%

Im going to reach out to my ISP to try and rule out any issues in my area. Historically they have been very stable but see these other alerts

Feb 22 13:11:00	dpinger	67738	WAN_DHCP nnn.20.25.1: Alarm latency 7258us stddev 376us loss 33%
Feb 22 13:11:10	dpinger	67738	WAN_DHCP nnn.20.25.1: Clear latency 20093us stddev 51225us loss 0%
Feb 22 13:11:12	dpinger	52955	send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 1 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr nnn.20.25.1 bind_addr nnn.20.27.213 identifier "WAN_DHCP "
Feb 22 18:19:57	dpinger	52955	WAN_DHCP nnn.20.25.1: Alarm latency 8521us stddev 1857us loss 21%
Feb 22 18:21:21	dpinger	52955	WAN_DHCP nnn.20.25.1: Clear latency 8512us stddev 2109us loss 15%
Feb 22 21:05:47	dpinger	52955	WAN_DHCP nnn.20.25.1: sendto error: 55
Feb 22 21:38:44	dpinger	52955	WAN_DHCP nnn.20.25.1: sendto error: 55
Feb 23 12:40:43	dpinger	52955	WAN_DHCP nnn.20.25.1: Alarm latency 9395us stddev 2343us loss 21%
Feb 23 12:41:14	dpinger	52955	WAN_DHCP nnn.20.25.1: Clear latency 9150us stddev 2720us loss 19%
Feb 23 12:43:14	dpinger	52955	WAN_DHCP nnn.20.25.1: Alarm latency 9256us stddev 2582us loss 21%
Feb 23 12:43:28	dpinger	52955	WAN_DHCP nnn.20.25.1: Clear latency 9462us stddev 2707us loss 19%

All observed since the 21.02 release. I will reach out to my ISP to try and rule them out.

FeatherKing

no known issues on the ISP side (so they say). I just loaded 21.02-p1. I will see if that is any different.

stephenw10

Any observed difference there?

FeatherKing

@stephenw10 tldr: better, but im not convinced it is update related.

since loading 21.02 on Feb 19 I have had 119 occurrences of packet loss (gathered from the Gateway log). Since loading 21.02-p1 I have only had 4. Feb 21-23 were not as severe as Feb 19/20. I am still inclined to pin this on my ISP, but things are better for now.