pfSense not responding to some ARP requests
-
I have a network with pfSense running on a 4 NIC computer. 2 of those ports are joined into a load balance LAGG which connects to a TL-1016PE switch. That switch then has single cable links to two other switches, "sw1" (a TL-SG108PE) and "swbench" (a TL-SG105E), and a pair of LAG'ed cables to another TL-SG1016PE. I have several VLANs/subnets in use, and all are trunked from the pfSense LAGG interface through all of these switches.
Recently I setup Uptime Kuma to monitor the uptime of a bunch of devices on my network, including the management IP of the switches. It sends pings from its location in VLAN 20 to the switch management IPs in VLAN 1. These checks work for everything in my network, including both the TL-1016PE switches. However, "sw1" and "swbench" frequently report missed ping responses. It happens multiple times per hour and can last anywhere from a 20 seconds to 5 minutes before the pings from Uptime Kuma start working again. When the switch management interfaces are not responding to Uptime Kuma, other traffic flowing through the switches (to devices connected to them) is not impacted. Additionally, the devices in VLAN 1 or in other VLANs other than the one where Uptime Kuma lives, can ping the switches successfully.
I started thinking this was an issue with Uptime Kuma or my VLAN config, but after getting packet captures from pfsense, and a device inline at every cable between swbench and pfsense, I see a clear picture of what is happening. The impacted switches have to send their ping response back through pfsense to Uptime Kuma since they are in different subnets. The switches send pereiodic ARP requests to discover the MAC of their gateway (pfsense) and most of the time pfsense sends back a ARP response. However, when the issue occurs, I see the ARP request come in to pfSense (with a pcap taken from pfsense) but no response be sent back to the switch. The switch then retries multiple times but pfsense never replies. At that point, the switch doesn't know how to address the ping response packet so it never gets sent. Eventually (sometimes 30 seconds, some times 5 minutes) pfsense will send a arp response to the switch and the ping health checks start working again. To further confirm the lack of arp response is the issue, I setup a script running on a computer in that same subnet to basically spam the subnet with arp responses (one every 15 seconds) pointing the switch to the MAC address of pfsense. That has "resolved" the issue. As soon as I stop that script, the issue will return within minutes.
In totality the evidence I have shows that when the issue happens:
- Ping requests are making it from Uptime Kuma all the way to the switch management IP
- A ping response is never sent out from the switch management IP
- The switch continues to send ARP requests asking for the MAC of its configured gateway (pfSense)
- That request is making it all the way to pfSense
- pfSense doesn't send an ARP response
Given other traffic keeps flowing through the switch while this issue is going on, the fact that pings coming from a host in the same subnet are responded to, and the packet captures we have of the ARP requests making it all the way to pfsense, I think we can rule out the switch being stalled/frozen, cabling issues, VLAN/subnet configuration issues, etc. All I'm left with is for some reason pfSense is just not responding to these ARP requests.
Complicating things, this only happens to 2 of my switches -- I have dozens of other Uptime Kuma ping based monitors running that never see this behavior. Further, it doesn't happen consistently -- pfsense answers some of the ARP requests from these switches, but occasionally just doesn't. I can also see that pfsense is sending other ARP responses during the issue, so pfSense isn't frozen. Over a 22-minute capture I did, pfSense answered 11.8% of swbench's gateway ARP requests and 56.5% of sw1's, while every other device on the same VLAN — got 100%.
Anyone have thoughts on what could be causing this, or how to troubleshoot further?
Running pfSense CE 2.8.1-RELEASE. I have Suricata, pfblockerNG, and other packages installed -- though given we are seeing issues with layer 2 ARP, I don't think any of those could have an impact.
-
said in pfSense not responding to some ARP requests:
Additionally, the devices in VLAN 1 or in other VLANs other than the one where Uptime Kuma lives, can ping the switches successfully.
Correction, only devices in VLAN 1 can ping the switches successfully.
-
@rtadams89 Try disconnecting one of the links in the LAGG from pfsense to the root switch. It sounds like LAGG misconfiguration
-
@keyser I did try that while one of the switches was experiencing the issue, and at that time the switch did start responding, however I think that was just coincidence as I have not been able to reproduce. Additionally, I have left one of the LAGG cables disconnected for hours and have seen the same failures with the downstream switches while that cable remained unplugged. I then plugged that cable back in and unplugged the other in the LAGG (to ensure it wasn't a cable or specific physical port issue) and the problems persisted.
-
@rtadams89 Try dissolving the LAGG in both ends and run the link as a simple VLAN trunk single link. Then we know if it's LAGG related or something else.
-
@keyser I have a bunch of VLANs defined on that LAGG interface — can I delete the aggregation without breaking or having to recreate all those VLANs? Ultimately, I need the LAGG, so this wouldn’t be a “fix”, but I can see it being a diagnostic step.
-
@rtadams89 I'm not familiar with those switches, but VLAN definitions is always "switchwide", so they will still be there even if you remove the LAGG. But depending on the switch it can be a lot of work to tag the VLANs to the port you setup for single link.
Alternative: Leave the lag, unplug it and don't use it. Just tag the VLANs to a different available port and use that for the test uplink
-
@keyser I’m talking about on the pfsense side — I have two ports used for LAN connectivity. Right now they are joined into a LAGG, and “on top of” that LAGG interface are a bunch of VLANs. If I delete the LAGG in pfsense to return to just 2 independent physical ports, I’m worried I will have to delete and recreate the virtual vlan interfaces on top of those physical interfaces. No?
-
@rtadams89 Ahhh, yeah, that takes a little more work. Remove one interface from the LAGG (so it only contains one link).
create the same VLAN tags (numbers) on that interface (leaving the ones you have on the LAGG.
Reassign your interfaces to the new single NIC.vlantag from LAGG.vlantag. Move the link.NOTE: More often than not I have had to restart the firewall to make interface assignments like that work properly, so don't get discouraged if it does not work initially
-
@keyser I’m hesitant to do this. This is a lot of config tired to the existing LAGG interface and its associated VLANs. Hundreds of firewall rules, suricata rules, pfsense and package configurations….
I’m also thinking the other tests have ruled out the LAGG as an issue: running with only one cable/port pair connected, seeing the arp request packets are making it all the way to pfsense based on the pfsense pcap.
Is there a reason to suspect the LAGG config in pfsense is the source of issue — and if it turns out to be, is there a path to fixing it?
-
@rtadams89 I'm not saying the LAGG pfsense config is the culprit, I'm saying the switch <-> pfsense LAGG interaction is the culprit. That is usually the case in scenarios/symtomps like yours. But the problem can be on either side in terms of config inconsistency (usually the problem).
But it can be - albeit theoretical - hardware/LAGG implementation inconsistency. -
@keyser ahh I see what you are saying. If just disconnecting one cable isn’t enough, I should be able to delete 1 of the physical interfaces out of the LAGG assignment on the pfsense side, leaving a LAGG with just one physical interfaces assigned (will pfsense let me do that?), break the LAGG config on the switch, and just keep the cable connecting the pfsense LAGG to the switch in place. Does that plan work?
-
@rtadams89 said in pfSense not responding to some ARP requests:
@keyser ahh I see what you are saying. If just disconnecting one cable isn’t enough, I should be able to delete 1 of the physical interfaces out of the LAGG assignment on the pfsense side, leaving a LAGG with just one physical interfaces assigned (will pfsense let me do that?), break the LAGG config on the switch, and just keep the cable connecting the pfsense LAGG to the switch in place. Does that plan work?
Exactly, that should work. But using the LAGG with only one link defined is still a LAG, and likely exibits the same problem. So you need step 2 by defining the VLAN tags on the free’d up link, and use that for uplink instead.
-
@keyser There is no practical way for me to completely eliminate the LAGG (too many things tied to the LAGG virtual interface on the pfsense side), but I did the following -- disconnected one of the two Ethernet cables previously part of the LAGG. Removed all the LAGG configuration on the switch side. As such, the Switch thinks it just has a single uplink to pfSense, and pfsense sees only one active physical link in it's LAGG group to the switch. The problem persists. I can still see the ARP request packets making it to pfsense (based off a pfsense packet capture on that interface) and no ARP reply going back out.
As I think through this, the fact that a tcpdump running on pfsense shows the ARP request coming in eliminates any hardware issues (on the switch or pfsense side). Is it possible pfsense is deliberately rejecting the request (due to some sort of rate limit or other reason)? Where would that show up in logs? Is it possible pfsense is for some reason replying on the wrong interface? Is there a way to do a packet capture on all interfaces simultaneously to see if this is the case?
-
@rtadams89 Well, i can’t think of a situation where pfSense would answer on another interface (except if you configured a bridge of the Lagg with another interface).
The reason I’m asking for the removal of the LAGG is because those does have a setup that causes them to ignore broadcasts on one interface in certain situations…Anyhow, I have no other suggestions - just the observation that I have never seen or heard of a situation where pfSense would intermittently not reply to ARP on an interface. I assume you do not have Ethernet Filtering active?
-
I've been digging into this over the last couple of day. The short answer is, this isn't pfsense. It seems that 3 of my switches (across 2 different manufactures) all have the same issue. They frequently send out arp requests for the gateway IP from their management IP, but tagged with a vlan tag (when they should be untagged). I know 2 of these switches (TP-Link brand) use realtek chipsets. I don't know about the other but I may pop it open to see if I can identify the chip. My best guess is this is just an issue in the Realtek chipset or the Realtek sdk/example firmware I'm guessing all these are using. The only think that still has me a little surprised is that I cannot find anyone else reporting this same issue. With how popular these switches, and the underlying chipsets are, I would think this would be more widely noticed and talked about.
-
@rtadams89 tplink and switches not a good mix.. Been quite a few years now - but they had an issue for the longest time where they would not allow you to remove vlan 1, even if you put a port in vlan X as access, ie native not tagged. It would still leave vlan 1 on there also as untagged.
Took them a few years to fix - they kept saying that was normal operation. And when they did finally fix it for v3 of the switch - they didn't backport it say v2 and v1 of the switches. Some smart person here posted a way to make the v2 switches take the v3 firmware - and now the switch reported it was v3 hardware ;)
What tag is it sending for its management IP in the arp? What is the 3rd model of switch are you seeing this in? Is it maybe Omada? Which is just different brand of tp-link.
Personally I would just trash those switches - that company clearly doesn't understand how tags and vlans work IMHO..
-
@johnpoz I see this happening on a TL-SG105E 5.0, a TL-SG108PE, and a XikeStor SKS3200M-8GPY1XF (Chinese suspicious switch -- tested just to rule out TP-Link). All of these switches have 5-7 VLANs trunked to/through them. I see the management ARP requests getting tagged with any of the VLANs the switch processes, but seems to usually be the VLAN that is organically the most chatty on that switch. Not all ARPs are mis-tagged, some do come out without a VLAN tag at all as expected. I have not seen any get tagged with VLAN 1.
-
@rtadams89 if me I would just trash those shit switches.. But yeah going to have a problem if you are looking to arp the IP of an interface that is native (untagged) and switch is tagging the arp.
Privacy Policy · Cookie Policy