Netgate 2100 ARP problem after replugging WAN port
-
@mikedob i don't have any sonos in this particular VLAN. and I'm using RSTP, but cannot define a Spanning Tree per VLAN (using Unifi)
-
@stephenw10 i captured the pcap on my computer, connected to a switchport which is in the same VLAN.
It's strange that my client isn't seeing those replies, but somehow it was mapped correctly in the arp table (pfsense and client).
I cleared both arp tables but they kept on mapping correctly.But still there's a huge amount of arp requests going on and for some reason, the access reader device won't work until i reboot the pfsense
-
OK, then you wouldn't expect to see any of the ARP replies to any other device.
That doesn't explain why your client keeps ARPing for pfSense though.
Try running the pcap on pfSense and see what it's doing and seeing at the time.
Steve
-
@somerino give us a layout of your physical network and any possible trunk ports. A network loop of some kind or possibly a bad cable would cause things trying to get arp requests. Your slow network performance listed in the sonos thread is what has brought this thought to mind. I work as a professional AV installer technician and have seen improper sonos installs and other network loops cause this kind of issue. Clearing your arp table is only temporary. Search the net for sonos and network storms. It's possible the sonos network is on a different vlan, but causing the trunk port to be over congested or you have another issue
-
@somerino as an example my personal network is. Fiber modem to luxul router then to netgate 6100 wan and USW-PRO-24-POE. 2 ports of switch have access points with 4 ssid, 1 said is passed back to the luxul with vlans. The other 3 are for use of vlans from pfsense. I also have a main trunk with 10 gig sfp between 6100 and switch. Due to careful vlan setup I have avoided loops in the network, and don't have the floods of arp requests. I also use RTI Control equipment, lutron lighting smart hubs , Sonos, Denon HEOS, Nuvo Wireless, multiple tv, printers, and starting to setup Savant. Some of the networks I've setup for large homes have been aruba 6300m switches and aruba 535 aps. Starting to use pfsense because of demand from the customers for security and multiple networks. I will be honest I have not done much with firewall rules as I'm trying to get equipment working first and then lock it down as needed.
-
@mikedob
First of all I want to thank you and @stephenw10 for trying to help me with this issue! I really appreciate it.Network:
I've a router from my ISP (it must be this device) connected to my netgate 2100 via a CAT7 ethernet cable. Then I've connected the Netgate 2100 to a USW-16-PoE which is the root switch.
From there on the switch is connected to multiple sub switches (USW-16-Lite-PoE, USW-8 etc). In the port profile I've created a Trunk_Switch profile which tags the necessary VLANS and as native I'm using an unused VLAN.
I'll exclude the AP from my description, since the problem occurs on wired devices.I have a VLAN for clients & printers, chromecast, voip and the mentioned access readers which have a problem. Those are devices from the company Gantner which can be used to control access through a door. I doubt that the error is caused on their side, because some devices work and some don't after replugging the WAN port. I've double checked the firmware and configuration of those devices and I couldn't find any error. As you mentioned, maybe it's a cabling issue, but if that's the case. I wonder why everything works fine, after rebooting the netgate 2100?
-
Do you have more than one gateway defined in the 2100? Is the WAN gateway set as default rather than set to auto?
If you do have multiple gateways and the default is set to auto the system default gateway will change to some other gateway when you disconnect the WAN and may not change back. That can obviously be a routing issue.
Otherwise run a pcap on the 2100 LAN when it's in the failed state and see what ARP messages it's seeing.
Steve
-
It's set as default. I've a gateway WAN_DHCP and a OpenVPN gateway to somewhere else.
Thanks for this tip, I'll set it to auto, to see if this resolves the issue.I'll create a PCAP of the 2100 LAN when it's in the failed state.
-
You need to set the default IPv4 gateway to 'WAN_DHCP', not auto (which is the default setting).
It's usually not an issue for OpenVPN gateways since they go down at the same time as the WAN. But definitely worth making that change anyway.
Steve
-
@stephenw10
Oh glad that you just said it...
I've set it to auto and the default was set to a VPN tunnel... -
Ah, in that case you may have set the specifically as part of the VPN setup to force all traffic over the VPN?
Usually that's OK. This could be unrelated to what ever's causing the apparent ARP issue.
Steve
-
@somerino ok network storms sometimes take time to build. The reason why I went after the sonos is because they can be bridging 1 network to a different network from its wireless. It's best to make sure all the sonos equipment is on the same and using an ssid you have control of. Also on some sonos equipment there are multiple ports. They are small 2 port switches, I advise only using 1
I've taken a 2nd look at the packets you posted. And see multiple devices asking for the mac address of a device, but the first part looks like it's looped, with a response from the device in question. Then other devices are asking and getting no response.
Firewall rules?
-
@stephenw10
I've run a PCAP on a self-hosted pfsense that had the same error state.
IntelCor_36:fb:39 (Pfsense)
Ubiquiti_d2:1a:a4 (UniFi Switch directly connected to the pfsense)What I've noticed is that the issue I've mentioned above isn't only related to a device from one vendor.
-
I don't have any sonos in this case. But in my other post about sonos. I've seen on my UniFi Switch on a sonos port that it was declared as downlink to my core switch, which is absolute bs. That caused a loop
-
Mmm, this starts to look like the switch doing something odd. Some loop/storm prevention setting maybe?
You can see pfSense is responding to every ARP request it sees but it appears the requester is not seeing that response. If that's the directly connected switch it's hard to see how it could be failing to see it....
The only other possibility is the driver/NIC doing something odd to the packet before it physically leaves. But you're seeing that across two different NIC types on two different architectures.Steve
-
Steve I've simplified the network and there's only one switch connected to the pfsense. zero possibility for a loop. storm control is also disabled
The pfsense is still spamming ARP request, it never receives an answer, because the device with the IP: 192.168.70.4 is offline.Is this a normal behaviour?
-
@somerino said in Netgate 2100 ARP problem after replugging WAN port:
The pfsense is still spamming ARP request, it never receives an answer, because the device with the IP: 192.168.70.4 is offline.
Yes, that's fine. If you have something referencing it in the config and pfSense is trying to send traffic to it, an internal DNS server for example, it will keep ARPing for it until it responds.
Steve
-
@somerino sorry for not understanding that this is a different system than the slow sonos system you were working with. The original arp scan and the most recent ones you posted look normal. Like stephenw10 has said. My personal system has 2 IoT controllers ARPing for devices I shifted to a different VLAN 3 weeks ago. You probably don't have a ARP problem but may have something else wrong with your Gantner equipment.
For your Gantner equipment failing to work after loss of connection on the wan port makes me wonder what kind of cloud or internet or VPN connection needs to be reestablished. I would look at its full normal communication all packets not just ARP.
For example I'm currently researching the packet process for the DIAL protocol for multiscreen casting. It seems to be a different flavor of ssdp upnp and mdns that Avahi and pimd don't resolve -
What I've noticed is that after plugging the WAN port back in. The OpenVPN service works, but it's in a half functioning state:
Sorry for the blurry picture the status says: "Waiting for response from peer"
But still it's shown online:
Most of the network still works in this state except my access readers (gantner). Only through manually restarting the VPN tunnel, the problems gets resolved and I can see traffic on the VPN status page.
I'm using the Peer To Peer (SSL/TLS) server mode and I really haven't experienced such an issue before upgrading the netgate 6100 (other end of the connection) to 22.05
-
Hei mike, I think you're right, it's not an ARP issue.
I've captured the traffic between the server and the gantner device and it literally only sends a phantom byte and closes the connection:
Even weirder, sometimes I can only see the tcp handshake without closing the connection.
The problem is so frustrating. I've mentioned below, that It might be VPN related. But I can't figure out, why everything else works through the VPN, except those gantner readers.
I've found a work-around by restarting the VPN manually. There comes the next problem. I've to do it everyday, because for some other silly reason, the VPN tunnel automatically restarts at night a few times.Jul 27 22:12:42 WHQ-FW01 check_reload_status[414]: Reloading filter Jul 27 22:12:43 WHQ-FW01 php-fpm[51938]: /rc.openvpn: Gateway, NONE AVAILABLE Jul 27 22:12:43 WHQ-FW01 php-fpm[51938]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use BUELACH_VPNV4. Jul 27 22:26:00 WHQ-FW01 sshguard[87099]: Exiting on signal. Jul 27 22:26:00 WHQ-FW01 sshguard[39533]: Now monitoring attacks. Jul 27 22:35:46 WHQ-FW01 rc.gateway_alarm[38160]: >>> Gateway alarm: BUELACH_VPNV4 (Addr:10.0.0.2 Alarm:1 RTT:32.705ms RTTsd:2.141ms Loss:22%) Jul 27 22:35:46 WHQ-FW01 check_reload_status[414]: updating dyndns BUELACH_VPNV4 Jul 27 22:35:46 WHQ-FW01 check_reload_status[414]: Restarting IPsec tunnels Jul 27 22:35:46 WHQ-FW01 check_reload_status[414]: Restarting OpenVPN tunnels/interfaces Jul 27 22:35:46 WHQ-FW01 check_reload_status[414]: Reloading filter Jul 27 22:35:47 WHQ-FW01 php-fpm[51938]: /rc.openvpn: Gateway, NONE AVAILABLE Jul 27 22:35:47 WHQ-FW01 php-fpm[51938]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use BUELACH_VPNV4. Jul 27 22:38:04 WHQ-FW01 rc.gateway_alarm[33071]: >>> Gateway alarm: BUELACH_VPNV4 (Addr:10.0.0.2 Alarm:0 RTT:23.979ms RTTsd:1.969ms Loss:5%) Jul 27 22:38:04 WHQ-FW01 check_reload_status[414]: updating dyndns BUELACH_VPNV4 Jul 27 22:38:04 WHQ-FW01 check_reload_status[414]: Restarting IPsec tunnels Jul 27 22:38:04 WHQ-FW01 check_reload_status[414]: Restarting OpenVPN tunnels/interfaces Jul 27 22:38:04 WHQ-FW01 check_reload_status[414]: Reloading filter Jul 27 22:38:06 WHQ-FW01 php-fpm[375]: /rc.openvpn: Gateway, NONE AVAILABLE Jul 27 22:38:06 WHQ-FW01 php-fpm[375]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use BUELACH_VPNV4.Jul 27 22:38:04 WHQ-FW01 rc.gateway_alarm[33071]: >>> Gateway alarm: BUELACH_VPNV4 (Addr:10.0.0.2 Alarm:0 RTT:23.979ms RTTsd:1.969ms Loss:5%) Jul 27 22:38:04 WHQ-FW01 check_reload_status[414]: updating dyndns BUELACH_VPNV4 Jul 27 22:38:04 WHQ-FW01 check_reload_status[414]: Restarting IPsec tunnels Jul 27 22:38:04 WHQ-FW01 check_reload_status[414]: Restarting OpenVPN tunnels/interfaces Jul 27 22:38:04 WHQ-FW01 check_reload_status[414]: Reloading filter Jul 27 22:38:06 WHQ-FW01 php-fpm[375]: /rc.openvpn: Gateway, NONE AVAILABLE Jul 27 22:38:06 WHQ-FW01 php-fpm[375]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use BUELACH_VPNV4.
