SG-3100 Packet Loss on LAN interface

phires · Oct 21, 2019, 3:28 PM

Hi everyone,

I've replaced my self build (some AMD machine) with a SG-3100. I never had issues with my build, but felt it would be right to support netgate by buying official hardware.
Nevertheless, I'm now having some trouble with it.
I can see quite a lot of packet loss just by pinging it (192.168.1.200 => 192.168.1.1):

--- 192.168.1.1 ping statistics ---
23 packets transmitted, 17 received, 26% packet loss, time 22001ms
rtt min/avg/max/mdev = 0.184/0.233/0.453/0.059 ms

Pinging any other IP in my network is not causing any issues, so I guess it's really an issue with the appliance (192.168.1.200 => 192.168.1.5):

--- 192.168.1.5 ping statistics ---
37 packets transmitted, 37 received, 0% packet loss, time 35993ms
rtt min/avg/max/mdev = 0.228/0.667/4.843/1.026 ms

Pinging FROM the SG-3100 to the network seems to be better with less packet loss, but still (192.168.1.1 => 192.168.1.200):

--- 192.168.1.200 ping statistics ---
100 packets transmitted, 99 packets received, 1.0% packet loss
round-trip min/avg/max/stddev = 0.211/0.556/4.579/0.915 ms

There is just one single switch in between (Netgear GS716Tv3) with no special configuration for the ports involved.

Here's what I've tried so far:

Lowering the MTU on the LAN interface to 1300 => no effect (not better, not worse)
Changing ports on the switch => no effect
Disable hardware checksum offload => no effect
Changed network cable => no effect

Hardware TCP segmentation offload & Hardware large receive offload were disabled from the beginning.
VLAN 802.1q is enabled, LAN 1 is still in VLAN 1, only LAN 4 is in VLAN 4084.

Is there anything else I can try to get the connection stabalized? Again, I've never had issues like that with my old hardware or any other device in my network, so it has to be something on the SG-3100.
What's strange to me is that it is worse when pinging TO the SG-3100 than pinging FROM it.

Thanks in advance,
Philipp

johnpoz · Oct 21, 2019, 4:04 PM

Well I have multiple sg3100 on multiple sites and can not duplicate this problem to any of them be the one local here, or the ones in other cities across our network

You sure your link is just not saturated?

What I would do for validation ping your your sg3100 from lan, while you sniffing on the lan interface on pfsense.. Do you actually see all the request.. So for example in your ping 192.168.1.1 you sent 23 - did pfsense actually see all 23 of those? And did it send a response or not?

phires · Oct 21, 2019, 4:17 PM

There is nothing going on in the network, the LAN interface is quite "bored" (<10kb/s)

Running a tcpdump on the SG-3100 yielded this result (removed MAC addresses):

18:08:49.527646 ethertype IPv4 (0x0800), length 74: 192.168.1.244 > 192.168.1.1: ICMP echo request, id 1, seq 1330, length 40
18:08:49.527684 ethertype IPv4 (0x0800), length 74: 192.168.1.1 > 192.168.1.244: ICMP echo reply, id 1, seq 1330, length 40
18:08:50.531768 ethertype IPv4 (0x0800), length 74: 192.168.1.244 > 192.168.1.1: ICMP echo request, id 1, seq 1331, length 40
18:08:50.531807 ethertype IPv4 (0x0800), length 74: 192.168.1.1 > 192.168.1.244: ICMP echo reply, id 1, seq 1331, length 40
18:08:56.524096 ethertype IPv4 (0x0800), length 74: 192.168.1.244 > 192.168.1.1: ICMP echo request, id 1, seq 1333, length 40
18:08:56.524131 ethertype IPv4 (0x0800), length 74: 192.168.1.1 > 192.168.1.244: ICMP echo reply, id 1, seq 1333, length 40
18:08:57.528229 ethertype IPv4 (0x0800), length 74: 192.168.1.244 > 192.168.1.1: ICMP echo request, id 1, seq 1334, length 40
18:08:57.528256 ethertype IPv4 (0x0800), length 74: 192.168.1.1 > 192.168.1.244: ICMP echo reply, id 1, seq 1334, length 40

As you can see, sequence 1332 is missing - this correlates to the lost packet on my client. So it seems as if the packets are not reaching the interface at all..?

I guess I'll further "dumb down" the setup by just directly connecting a device to the LAN1 port and check again.

phires · Oct 21, 2019, 5:52 PM

I did some further investigation. This is all pretty odd. I still have packet loss to the sg3100, but:

Connecting a laptop directly to LAN1 and pinging: no packet loss
Connecting a laptop to a "dumb" (unmanaged) switch, that is only connected to the laptop and sg3100: no packet loss
Connecting the unmanaged switch to the Netgear switch: packet loss, but only to the sg3100. No other device on the network has any issues, regardless which port or switch it is on. There is now even packet loss, when the traffic isn't going through the Netgear switch at all.

Something is causing strange effects when the sg3100 and the Netgear GS716Tv3 are physically connected to each other - regardless if there are other switches between them or not.

Before ramping everything down to factory defaults (causing all VLANs, LAGGs, etc. being removed): Any other idea?

phires · Oct 21, 2019, 6:07 PM

Final notice for today: Disabling 802.1q on the sg3100 solves the issue.
Something is not really working well with VLANs in my setup :(

johnpoz · Oct 21, 2019, 6:41 PM

I run vlans on all of my sg3100, ie 802.1q mode is enabled.. Have seen no issues.

When you sniff and pfsense does not see a packet - that is not on pfsense.. Seems your switch is not tagging stuff correctly maybe?