Can't ping the same IP from multiple devices
-
Ran into an interesting symptom. I kind of see it, but would like to understand whether it's a config issue or this is just how it is.
We have two routers, building and office, with double NAT. In this case the inner/office router is behind 1:1 NAT with a unique public IP. Both are on 24.11.
From the office LAN we've found we can't ping the same IP (8.8.8.8 here, but any IP) from more than one device at the same time. If we do, the second PC fails to get a response until the state on the building router closes.
The inner/office router lists both states:
10.y.y.108:1 -> 8.8.8.8:8 0:0 17 / 17 1020 B / 1020 B 10.y.y.104:1 -> 8.8.8.8:8 0:0 1 / 0 60 B / 0 BThe building router (unique subnet) only lists one:
10.x.x.42:1 -> 8.8.8.8:8From any two PCs:
ping -n 20 8.8.8.8
Pinging 8.8.8.8 with 32 bytes of data:
Reply from 8.8.8.8: bytes=32 time=12ms TTL=114
Reply from 8.8.8.8: bytes=32 time=12ms TTL=114
Reply from 8.8.8.8: bytes=32 time=12ms TTL=114
...ping -n 20 8.8.8.8
Pinging 8.8.8.8 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
...As soon as the "working" state expires the second devices gets its responses.
On the outer/building router I've tried setting the outbound NAT to random or static port, but again it's using 1:1.
This is not a problem with other types of connections, I'm assuming because the source port is random and not :1.
Is this just a side effect of double NAT and we simply haven't noticed this in the last couple decades?
In both cases the connection is using the LAN-to-any firewall rule.
-
@SteveITS sound a lot like FreeBSD Bug 283795 - ICMP echo requests from Windows hosts dropped when NAT'ed.
If I run
ping -e 1 8.8.4.4on two Linux machines on LAN, I hit the same issue.
If instead I runping -e 8 8.8.4.4the port (not sure that is the correct term) on WAN is set to random and it works.There is another bug (long) report (can't find it right now) in which it's mentioned that it is based on changes imported from OpenBSD. Unfortunately it sounded like ThisSense against TheOtherSense (which stated a possible reason for the bug) and it won't get fixed.
Addition: the other (fixed, long-read) bug Bug 280701 - FreeBSD-SA-24:05 fix breaks ICMP/ICMP6 states handling in pf firewall (ping, traceroute)
Addition 2: still broken on latewt 25.03-BETA -
@patient0 Interesting so we’re not crazy. It’s 1am so I’ll test more tomorrow. Not sure I tried pinging all our routers.
It’s something few would notice I expect but we of course diagnose connection issues for clients, and we have monitoring which pings everything in and on the way to our data center. Seems like we could trigger that by pinging a server ourselves.
-
@SteveITS said in Can't ping the same IP from multiple devices:
Not sure I tried pinging all our routers.
I don't run into the issue since I don't have any Windows clients. And it seems that
pingon Windows sets ICMP ID to 1 while on Linux it is set to a random id.FreeBSD (erroneously) set the WAN ICMP ID to the same as the source ICMP ID. And only if you set ICMP ID to 8 FreeBSD picks a random ID for the WAN to ping-destination. And since ICMP type ECHO_REQUEST numerical value is 8 ... mmhh, I'm no programmer.
-
@patient0 Interesting. Just checked with the OtherSense, it shows this too, but you already gave that away.
Edit: And checked with another Router, FreshTomato: behaved the same. -
@Bob-Dig said in Can't ping the same IP from multiple devices:
@patient0 Interesting. Just checked with the OtherSense, it shows this too, but you already gave that away.
Yes and it still does work with CE 2.7.2/FreeBSD 14. I think the feature/bug got introduced somewhere along FreeBSD 14.1/14.2 (in pf, ipfw doesn’t have the issue)
-
@Bob-Dig said in [Can't ping the same IP from multiple devices]
Edit: And checked with another Router, FreshTomato: behaved the same.
Mmh, that odd since it’s Linux based. I did check with VyOS back then and didn’t hit it. Have to recheck later this week.
-
@patient0 said in Can't ping the same IP from multiple devices:
that odd since it’s Linux based
But the host was Windows, so I guess, it is somewhat expected. It probably was nice of *Sense, that they had a "mitigation" for this Windows behavior.
-
Yeah if you test from something that's not Windows you'll probably find it works fine. For some reason Windows uses the same ID for all pings. So if you have 1:1 NAT (or static ports outbound NAT) then only one internal system can open a unique state. Linux uses incremental IDs. BSD uses random IDs.
-
@stephenw10 said in Can't ping the same IP from multiple devices:
So if you have 1:1 NAT (or static ports outbound NAT)
It doesn't seem to be related to that. It has worked in the past (according to the thread) and doesn't right now.
-
@SteveITS FWIW it's also an issue pinging the outer/building router from our LAN, so doesn't need to go past the second router.
I suppose, it mostly only matters as a colossal time waster while troubleshooting, if you don't know of the bug, since it's probably uncommon to see it (more common, the larger the company, I suppose).
My first time, pinging from a Linux VM, then from Windows, the Windows pings failed. After that I can't seem to reproduce that failure.
At least that implies we maybe can't trigger a false failure on our monitoring if we happen to ping something at the same time as our monitoring software.
@stephenw10 We did have static outbound set on the outer/building router...the rule is timestamped 2018 so I don't recall now why I set that. :) But turning that off last night did not change the behavior. It was not set on the inner one.
-
1:1 NAT implies static ports so if you have that set you would still hit this.
-
@stephenw10 ah ha, did not realize/remember that.
Edit: OK so then few would see this. And in theory port forwarding all ports and configuring outbound NAT, for that VIP, would bypass it?
-
Yes it would. Though it only affects icmp from Windows so.... it mostly doesn't matter.
I remember that blowing my mind when I first saw it. Mostly because Linux clients were unaffected.
-
I seem to use this pic a lot lately.
-
@Bob-Dig said in Can't ping the same IP from multiple devices:
It doesn't seem to be related to that. It has worked in the past (according to the thread) and doesn't right now.
Actually I think you are right, as least as worded. I tried from two Windows PCs at home and can repro it there. Automatic outbound NAT, not static, no 1:1.
One can see where the first ping expired:
Pinging 8.8.4.4 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 8.8.4.4: bytes=32 time=18ms TTL=116
Reply from 8.8.4.4: bytes=32 time=22ms TTL=116
Reply from 8.8.4.4: bytes=32 time=19ms TTL=116
Reply from 8.8.4.4: bytes=32 time=19ms TTL=116
Reply from 8.8.4.4: bytes=32 time=21ms TTL=116
Reply from 8.8.4.4: bytes=32 time=21ms TTL=116 -
@SteveITS said in Can't ping the same IP from multiple devices:
I think you are right
I tried it with both Senses and with FreshTomato, without any special OutboundNAT, the outcome was every time the same.