Strange behaviour for ICMP (ping) rule on WAN interface
-
Yes, it's a problem.
em0 is the WAN and em8 is the Public LAN there right?The ARP requests from the gateway are not being passed to em8 which I would expect to see.
Where as the pfSense ARP requests for .5 are on both.
What appears on the VM when you are dong that?
-
@stephenw10
Yes, em0 is the Wan and em8 is the public LAN.This is what appears on the VM during the ping execution:
[root@test-hs01 ~]# ping y.y.y.1
PING y.y.y.1 (y.y.y.1) 56(84) bytes of data.
From y.y.y.5 icmp_seq=1 Destination Host Unreachable
From y.y.y.5 icmp_seq=2 Destination Host Unreachable
From y.y.y.5 icmp_seq=3 Destination Host Unreachable
From y.y.y.5 icmp_seq=4 Destination Host Unreachable
From y.y.y.5 icmp_seq=5 Destination Host Unreachable
From y.y.y.5 icmp_seq=6 Destination Host Unreachable
From y.y.y.5 icmp_seq=7 Destination Host Unreachable[root@test-hs01 ~]# ping y.y.y.2
PING y.y.y.2 (y.y.y.2) 56(84) bytes of data.Thanks,
Mauro -
Hello Stephen, so this solution (bridge) is not applicable in my case/scenario, right?
If yes, could you please suggest any other solution?
Many thanks for your support and patience.
Mauro -
Sorry I meant what appears in a pcap on the VM when you run those pings.
The other solutions here are:
Use a real routed subnet that you can then just apply to the Public LAN interface directly.
Use a VIP on WAN and NAT the traffic to the VM in a private subnet.If the VM needs to have a public IP directly and you do not have a routed subnet then bridging the interfaces is the only way to do it. In which case I'd suggest real hardware. Or maybe you could try hardware pass-thorugh for one the hypervisor NICs.
Steve
-
Hello Stephen, this is what happens on the VM during the ping (and while ARP request ):
tcpdump -i ens192
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
16:35:04.672983 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.f8:8e:a1:73:f9:81.8016, length 43
16:35:06.674441 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.f8:8e:a1:73:f9:81.8016, length 43
16:35:08.676482 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.f8:8e:a1:73:f9:81.8016, length 43
16:35:10.678418 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.f8:8e:a1:73:f9:81.8016, length 43
16:35:11.431985 IP test-hs01 > y.y.y.2: ICMP echo request, id 26911, seq 1, length 64
16:35:11.432228 ARP, Request who-has gateway tell test-hs01, length 28
16:35:12.431108 IP test-hs01 > y.y.y.2: ICMP echo request, id 26911, seq 2, length 64
16:35:12.433092 ARP, Request who-has gateway tell test-hs01, length 28
16:35:12.680009 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.f8:8e:a1:73:f9:81.8016, length 43
16:35:13.431096 IP test-hs01 > y.y.y.2: ICMP echo request, id 26911, seq 3, length 64
16:35:13.435085 ARP, Request who-has gateway tell test-hs01, length 28
16:35:14.431103 IP test-hs01 > y.y.y.2: ICMP echo request, id 26911, seq 4, length 64
16:35:14.437140 ARP, Request who-has gateway tell test-hs01, length 28
16:35:14.682674 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.f8:8e:a1:73:f9:81.8016, length 43
16:35:15.432145 IP test-hs01 > y.y.y.2: ICMP echo request, id 26911, seq 5, length 64
16:35:15.439085 ARP, Request who-has gateway tell test-hs01, length 28
16:35:16.432110 IP test-hs01 > y.y.y.2: ICMP echo request, id 26911, seq 6, length 64
16:35:16.433082 ARP, Request who-has y.y.y.2 tell test-hs01, length 28
16:35:16.433305 ARP, Reply y.y.y.2 is-at 00:0c:29:63:d5:85 (oui Unknown), length 46On pfsense .2 WAN interface (em0):
21:37:41.373299 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:42.373293 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:42.384679 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
21:37:43.373319 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:43.408734 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
21:37:44.373316 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:45.373317 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:46.373292 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:47.373311 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28On pfsense .2 LAN interface (em8):
21:37:41.373249 IP y.y.y.5 > y.y.y.2: ICMP echo request, id 26911, seq 151, length 64
21:37:41.373287 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:42.373242 IP y.y.y.5 > y.y.y.2: ICMP echo request, id 26911, seq 152, length 64
21:37:42.373281 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:42.384702 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
21:37:43.373270 IP y.y.y.5 > y.y.y.2: ICMP echo request, id 26911, seq 153, length 64
21:37:43.373307 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:43.408758 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
21:37:44.373266 IP y.y.y.5 > y.y.y.2: ICMP echo request, id 26911, seq 154, length 64
21:37:44.373304 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:45.373266 IP y.y.y.5 > y.y.y.2: ICMP echo request, id 26911, seq 155, length 64
21:37:45.373302 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28Regarding the alternative solution:
The other solutions here are:
Use a real routed subnet that you can then just apply to the Public LAN interface directly.
Use a VIP on WAN and NAT the traffic to the VM in a private subnet.You also said that:
It looks like you do have a routed subnet there, the /25 is routed to you over the /30.
But you can't use it as a routed subnet in pfSense because you have some other router upstream and the /25 is on the pfSense WAN directly.
In that situation you would have to bridge it to use the public IP on servers directly.Unfortunately, some months ago, we tried to remove the additional (our) upstream router and assign /30 address directly to pfsense WAN and /25 addresses to LAN without success (the network was unstable).
For this reason we are in this situation now.
So, this being the case, I have to choose the second alternative solution (VIP + NAT).Before closing this case, I would like to thank you very much for the time you dedicated to this case and the patience you have had.
I really appreciated. Many thanks for your support. -
Yeah, you can see in pcaps there pfSense is sending the ARP requests and bridging them between em8 and em8 but they never reach the VM.
I've done that sort of setup many times on bare metal installs and not had an issue so this has to something in the virtualisation.Since you do actually have a routed subnet I would try to go back to using that directly if you can. What was unstable about it?
Steve
-
Yeah, you can see in pcaps there pfSense is sending the ARP requests and bridging them between em0 and em8 but they never reach the VM.
I've done that sort of setup many times on bare metal installs and not had an issue so this has to something in the virtualisation.
It is really a mystery! I will investigate about the Intel 10GbE network card installed on the hypervisor. Maybe it is too old and it doesn't manage this kind of traffic...I don't know...
But if, as you said, both em0 and em8 are on the same layer 2, the problem could be different...Since you do actually have a routed subnet I would try to go back to using that directly if you can. What was unstable about it?
Unfortunately, the pfSense instance is in production and I can't change the entire configuration and firewall rules.
When we tried to assign /30 address directly to pfsense WAN and /25 addresses to LAN the network stopped working or it was working intermittently and slowly.Probably, I should set something else in the pfsense configuration to make it working.
-
stephenw10
Good evening, I would like to ask you two last questions before closing this case:
In your opinion, is the problem more related to some hypervisor settings or is it related to the problem that the two interfaces (WAN and public LAN) are on the same layer 2?If I decide to configure a routed subnet as you suggested, should I disable the "outbound NAT" to make it working?
If yes, outbound NAT should no more be used for some other needs related to the other LANs defined on pfSense, right?Thank you,
Mauro -
It's the combination of the bridged interfaces and running as a VM. Either will work fine by themselves. They can probably work fine together if you have the correct settings in the hypervisor.
You can disable outbound NAT for only the routed subnet and keep it for the others.
Steve
-
@stephenw10 thank you very much, Stephen. You really helped me to understand a lot go things. Have a great day. See you in the next topic :)
