Strange behaviour for ICMP (ping) rule on WAN interface

mauro.tridici

Hello Stephen,

something changed after moving from E1000 to VMXNET3 nics type.
The problem seems to be still here, but I think that your professional experience could help to resolve the issue.

In order to do some test without involving the pfSense instance that is in production, I preferred to perform the following steps:

clone of the production pfSense instance;
change of the nic type from E1000 to VMXNET3 (only for the WAN and "Public LAN" interfaces);
update of the pfsense interfaces names (vmx0 for the WAM, vmx2 for the public LAN);
assignment of a new public IP (belonging to the same public subnet mentioned in the messages above) to the WAN interface (y.y.y.15/25);
disconnection of every virtual interface defined on the pfsense instance (except for WAN and public LAN).

After that, I started doing some test.

Ping from VM (IP .5) to pfSense (IP .15):

output on VM:
ping y.y.y.15 (ping seems waiting for the answer)

output on PFSENSE:
[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: tcpdump -i vmx2 host y.y.y.5
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmx2, link-type EN10MB (Ethernet), capture size 262144 bytes
15:08:35.989761 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
15:08:36.780956 IP y.y.y.5 > y.y.y.15: ICMP echo request, id 19822, seq 65, length 64
15:08:36.780982 ARP, Request who-has y.y.y.5 tell y.y.y.15, length 28
15:08:37.004806 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
15:08:37.780931 IP y.y.y.5 > y.y.y.15: ICMP echo request, id 19822, seq 66, length 64
15:08:37.780952 ARP, Request who-has y.y.y.5 tell y.y.y.15, length 28
15:08:38.028828 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
15:08:38.780927 IP y.y.y.5 > y.y.y.15: ICMP echo request, id 19822, seq 67, length 64
15:08:38.780965 ARP, Request who-has y.y.y.5 tell y.y.y.15, length 28
15:08:39.780917 IP y.y.y.5 > y.y.y.15: ICMP echo request, id 19822, seq 68, length 64
15:08:39.780966 ARP, Request who-has y.y.y.5 tell y.y.y.15, length 28
15:08:40.780948 IP y.y.y.5 > y.y.y.15: ICMP echo request, id 19822, seq 69, length 64
15:08:40.781007 ARP, Request who-has y.y.y.5 tell y.y.y.15, length 28

Now, the ICMP request can reach PFSENSE, but PFSENSE didn't reply to this request.
It seems that something is blocking this kind of traffic.

Ping from VM (IP .15) to pfSense (IP .5):

[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: ping y.y.y.5
PING y.y.y.5 (y.y.y.5): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down

ARP TABLE on PFSENSE:

[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: arp -a
? (192.168.120.1) at 00:0c:29:db:d8:06 on vmx3.192 permanent [vlan]
? (10.0.0.1) at 00:0c:29:db:d8:06 on vmx3.10 permanent [vlan]
? (192.168.44.1) at 00:0c:29:db:d8:d4 on lagg0.44 permanent [vlan]
? (192.168.43.1) at 00:0c:29:db:d8:d4 on lagg0.43 permanent [vlan]
? (192.168.40.1) at 00:0c:29:db:d8:d4 on lagg0.40 permanent [vlan]
? (192.168.34.1) at 00:0c:29:db:d8:d4 on lagg0.34 permanent [vlan]
? (192.168.33.1) at 00:0c:29:db:d8:d4 on lagg0.33 permanent [vlan]
? (192.168.32.1) at 00:0c:29:db:d8:d4 on lagg0.32 permanent [vlan]
? (192.168.31.1) at 00:0c:29:db:d8:d4 on lagg0.46 permanent [vlan]
? (192.168.30.1) at 00:0c:29:db:d8:d4 on lagg0.30 permanent [vlan]
? (y.y.y.5) at 00:0c:29:2b:ee:a6 on vmx2 expires in 1109 seconds [ethernet]
? (y.y.y.15) at 00:0c:29:db:d8:c0 on vmx0 permanent [ethernet]
? (y.y.y.1) at 00:0c:29:02:f1:99 on vmx0 expires in 1200 seconds [ethernet]
? (192.168.100.1) at 00:0c:29:db:d8:f2 on em4 permanent [ethernet]
pfSense_LAN_CMCC.home.arpa (192.168.240.2) at 00:0c:29:db:d8:de on em2 permanent [ethernet]

ARP TABLE on VM:

gateway (y.y.y.1) at <incomplete> on ens192
? (y.y.y.15) at 00:0c:29:db:d8:10 [ether] on ens912

It seems that the bridge is allowing the traffic from VM to pfsense and is blocking the traffic from pfsense to VM.
Do you have some other idea about the cause of this issue?

Many thanks in advance,
Mauro

mauro.tridici

It seems that pfsense is trying to reach the .5 using a different route.
Maybe it is trying to do it via the upstream gateway .1. But VM .5 is not "on internet", it is behind the firewall itself.

stephenw10

pfSense there is sending an ARP request for .5 every second. So probably each time it tries to reply to a ping. That implies it never sees a response. And in fact the pcap never shows the VM responding to those requests.
The odd thing there is that the pfSense ARP table appears to have an entry for .5 even though it's still sending queries for it. Was that taken after the pings stopped?

Steve

mauro.tridici

@stephenw10
No it was taken during the ping.
This is what I see when the ping is stopped:

[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: tcpdump -i vmx2 host y.y.y.5
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmx2, link-type EN10MB (Ethernet), capture size 262144 bytes
15:31:05.356375 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
15:31:06.382707 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
15:31:07.406663 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
15:31:13.187718 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
15:31:14.190695 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
15:31:15.214728 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
15:31:22.649162 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
15:31:23.662695 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
15:31:24.686714 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46

stephenw10

Right but you still see an entry in the ARP table in pfSense for .5?

mauro.tridici

@stephenw10 I just checked, I don't see any entry for .5 :-(
Do you think that I should give up?

Is my opinion wrong?

It seems that pfsense is trying to reach the .5 using a different route.
Maybe it is trying to do it via the upstream gateway .1. But VM .5 is not "on internet", it is behind the firewall itself.

stephenw10

This looks like a layer 2 issue. pfSense is sending ARP requests and the VM never replies, so it's probably not seeing them. A pcap on the VM would confirm that.

mauro.tridici

I still don't understand why everything works if I assign a static IP to the "public LAN" interface... connectivity stops working as soon as the bridge is enabled.
Why VM stops sending ARP replies as soon as I change the IP address?

I don't want to disturb you again, sorry.
I'm doing these questions to myself :)

stephenw10

Is the VM actually seeing the ARP requests? More likely the virtual interface is not sending them from pfSense because to do so it has to use the wrong MAC address. That's why it needs to be in promiscuous mode. Likely something has to be set in VMWare to allow that for the hypervisor side.

https://kb.vmware.com/s/article/1004099

mauro.tridici

Thank you, Stephen.
The Wan and "Public LAN" interfaces (and related switches) are in promiscuous mode. Also the VM interface and vswitch is in promiscuous mode.

If I run "tcpdump -I ens192" on the VM, I can see only the STP lines:

3: ens192: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:0c:29:2b:ee:a6 brd ff:ff:ff:ff:ff:ff
inet y.y.y.5/25 brd 90.147.177.127 scope global noprefixroute ens192
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe2b:eea6/64 scope link
valid_lft forever preferred_lft forever

[root@test-hs01 ~]# tcpdump -i ens192
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
15:37:24.676212 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.f8:8e:a1:73:f9:81.8016, length 43
15:37:26.678015 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.f8:8e:a1:73:f9:81.8016, length 43
15:37:28.679573 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.f8:8e:a1:73:f9:81.8016, length 43
15:37:30.681543 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.f8:8e:a1:73:f9:81.8016, length 43
15:37:32.683314 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.f8:8e:a1:73:f9:81.8016, length 43
15:37:34.684944 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.f8:8e:a1:73:f9:81.8016, length 43

stephenw10

pfSense will only ARP for it when it's trying to send traffic to it. So when either pfSense is trying to ping the VM or when the VM is pinging pfSense and it's trying to reply.

mauro.tridici

@stephenw10 yes, you are right, thank you.

If I make a ping from VM to PFSENSE, on pfSense I can see the correct ARP line:

[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: arp -a|grep y.y.y.5
? (y.y.y.5) at 00:0c:29:2b:ee:a6 on em8 expires in 1180 seconds [ethernet]
? (y.y.y.5) at (incomplete) on em0 expired [ethernet]

But, I also noticed that there is an incomplete ARP line related to .5 IP.
And this line contains some reference to the em0 interfaces (that is the WAN interface for pfSense).

Is it normal or should I investigate on it?

stephenw10

Well they are in the same layer 2 so the normal things don't really apply there. Do you see the ARPs or ping replies in a pcap on either device?

mauro.tridici

@stephenw10

Do you see the ARPs or ping replies in a pcap on either device?

I'm not sure I have correctly got it.
Do you mean that I should run a pcap on both the interfaces (em0 and em8) while a ping from the VM is running?

Thank you

mauro.tridici

I just captured pcap from pfsense on WAN (em0) and LAN (em8) interfaces while ping was running from VM to pfSense.
And yes, ARP requests are in both the pcap.

PINGING PFSENSE .2 FROM VM .5

[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: tcpdump -i em0 host y.y.y.5
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
23:37:25.533835 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
23:37:26.533813 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
23:37:27.533821 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
23:37:28.533818 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
23:37:29.533859 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
23:37:30.533777 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
23:37:31.533820 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
23:37:32.533848 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28

[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: tcpdump -i em8 host y.y.y.5
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em8, link-type EN10MB (Ethernet), capture size 262144 bytes
23:37:27.533765 IP y.y.y.5 > y.y.y.2: ICMP echo request, id 22232, seq 25, length 64
23:37:27.533809 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
23:37:28.533784 IP y.y.y.5 > y.y.y.2: ICMP echo request, id 22232, seq 26, length 64
23:37:28.533807 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
23:37:29.533783 IP y.y.y.5 > y.y.y.2: ICMP echo request, id 22232, seq 27, length 64
23:37:29.533851 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
23:37:30.533736 IP y.y.y.5 > y.y.y.2: ICMP echo request, id 22232, seq 28, length 64
23:37:30.533766 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28

PINGING ROUTER .1 FROM VM .5

[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: tcpdump -i em0 host 90.147.177.5
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 262144 bytes
23:42:34.447705 ARP, Request who-has 90.147.177.5 tell 90.147.177.1, length 46
23:42:35.066680 ARP, Request who-has 90.147.177.1 tell 90.147.177.5, length 46
23:42:35.471738 ARP, Request who-has 90.147.177.5 tell 90.147.177.1, length 46
23:42:36.069669 ARP, Request who-has 90.147.177.1 tell 90.147.177.5, length 46
23:42:36.805030 ARP, Request who-has 90.147.177.5 tell 90.147.177.1, length 46
23:42:37.071682 ARP, Request who-has 90.147.177.1 tell 90.147.177.5, length 46
23:42:37.807708 ARP, Request who-has 90.147.177.5 tell 90.147.177.1, length 46

[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: tcpdump -i em8 host 90.147.177.5
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em8, link-type EN10MB (Ethernet), capture size 262144 bytes
23:42:41.071582 ARP, Request who-has 90.147.177.1 tell 90.147.177.5, length 46
23:42:43.068626 ARP, Request who-has 90.147.177.1 tell 90.147.177.5, length 46
23:42:44.069600 ARP, Request who-has 90.147.177.1 tell 90.147.177.5, length 46
23:42:45.071622 ARP, Request who-has 90.147.177.1 tell 90.147.177.5, length 46
23:42:47.069638 ARP, Request who-has 90.147.177.1 tell 90.147.177.5, length 46
23:42:48.071598 ARP, Request who-has 90.147.177.1 tell 90.147.177.5, length 46

It is a problem, right? Do you think that there is a way to fix this behaviour?

Thanks,
Mauro

stephenw10

Yes, it's a problem.
em0 is the WAN and em8 is the Public LAN there right?

The ARP requests from the gateway are not being passed to em8 which I would expect to see.

Where as the pfSense ARP requests for .5 are on both.

What appears on the VM when you are dong that?

mauro.tridici

@stephenw10
Yes, em0 is the Wan and em8 is the public LAN.

This is what appears on the VM during the ping execution:

[root@test-hs01 ~]# ping y.y.y.1
PING y.y.y.1 (y.y.y.1) 56(84) bytes of data.
From y.y.y.5 icmp_seq=1 Destination Host Unreachable
From y.y.y.5 icmp_seq=2 Destination Host Unreachable
From y.y.y.5 icmp_seq=3 Destination Host Unreachable
From y.y.y.5 icmp_seq=4 Destination Host Unreachable
From y.y.y.5 icmp_seq=5 Destination Host Unreachable
From y.y.y.5 icmp_seq=6 Destination Host Unreachable
From y.y.y.5 icmp_seq=7 Destination Host Unreachable

[root@test-hs01 ~]# ping y.y.y.2
PING y.y.y.2 (y.y.y.2) 56(84) bytes of data.

Thanks,
Mauro

mauro.tridici

@stephenw10

Hello Stephen, so this solution (bridge) is not applicable in my case/scenario, right?

If yes, could you please suggest any other solution?

Many thanks for your support and patience.
Mauro

stephenw10

Sorry I meant what appears in a pcap on the VM when you run those pings.

The other solutions here are:
Use a real routed subnet that you can then just apply to the Public LAN interface directly.
Use a VIP on WAN and NAT the traffic to the VM in a private subnet.

If the VM needs to have a public IP directly and you do not have a routed subnet then bridging the interfaces is the only way to do it. In which case I'd suggest real hardware. Or maybe you could try hardware pass-thorugh for one the hypervisor NICs.

Steve

mauro.tridici

Hello Stephen, this is what happens on the VM during the ping (and while ARP request ):

tcpdump -i ens192
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
16:35:04.672983 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.f8:8e:a1:73:f9:81.8016, length 43
16:35:06.674441 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.f8:8e:a1:73:f9:81.8016, length 43
16:35:08.676482 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.f8:8e:a1:73:f9:81.8016, length 43
16:35:10.678418 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.f8:8e:a1:73:f9:81.8016, length 43
16:35:11.431985 IP test-hs01 > y.y.y.2: ICMP echo request, id 26911, seq 1, length 64
16:35:11.432228 ARP, Request who-has gateway tell test-hs01, length 28
16:35:12.431108 IP test-hs01 > y.y.y.2: ICMP echo request, id 26911, seq 2, length 64
16:35:12.433092 ARP, Request who-has gateway tell test-hs01, length 28
16:35:12.680009 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.f8:8e:a1:73:f9:81.8016, length 43
16:35:13.431096 IP test-hs01 > y.y.y.2: ICMP echo request, id 26911, seq 3, length 64
16:35:13.435085 ARP, Request who-has gateway tell test-hs01, length 28
16:35:14.431103 IP test-hs01 > y.y.y.2: ICMP echo request, id 26911, seq 4, length 64
16:35:14.437140 ARP, Request who-has gateway tell test-hs01, length 28
16:35:14.682674 STP 802.1w, Rapid STP, Flags [Learn, Forward, Agreement], bridge-id 8000.f8:8e:a1:73:f9:81.8016, length 43
16:35:15.432145 IP test-hs01 > y.y.y.2: ICMP echo request, id 26911, seq 5, length 64
16:35:15.439085 ARP, Request who-has gateway tell test-hs01, length 28
16:35:16.432110 IP test-hs01 > y.y.y.2: ICMP echo request, id 26911, seq 6, length 64
16:35:16.433082 ARP, Request who-has y.y.y.2 tell test-hs01, length 28
16:35:16.433305 ARP, Reply y.y.y.2 is-at 00:0c:29:63:d5:85 (oui Unknown), length 46

On pfsense .2 WAN interface (em0):

21:37:41.373299 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:42.373293 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:42.384679 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
21:37:43.373319 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:43.408734 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
21:37:44.373316 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:45.373317 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:46.373292 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:47.373311 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28

On pfsense .2 LAN interface (em8):

21:37:41.373249 IP y.y.y.5 > y.y.y.2: ICMP echo request, id 26911, seq 151, length 64
21:37:41.373287 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:42.373242 IP y.y.y.5 > y.y.y.2: ICMP echo request, id 26911, seq 152, length 64
21:37:42.373281 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:42.384702 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
21:37:43.373270 IP y.y.y.5 > y.y.y.2: ICMP echo request, id 26911, seq 153, length 64
21:37:43.373307 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:43.408758 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
21:37:44.373266 IP y.y.y.5 > y.y.y.2: ICMP echo request, id 26911, seq 154, length 64
21:37:44.373304 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28
21:37:45.373266 IP y.y.y.5 > y.y.y.2: ICMP echo request, id 26911, seq 155, length 64
21:37:45.373302 ARP, Request who-has y.y.y.5 tell y.y.y.2, length 28

Regarding the alternative solution:

The other solutions here are:
Use a real routed subnet that you can then just apply to the Public LAN interface directly.
Use a VIP on WAN and NAT the traffic to the VM in a private subnet.

You also said that:

It looks like you do have a routed subnet there, the /25 is routed to you over the /30.
But you can't use it as a routed subnet in pfSense because you have some other router upstream and the /25 is on the pfSense WAN directly.
In that situation you would have to bridge it to use the public IP on servers directly.

Unfortunately, some months ago, we tried to remove the additional (our) upstream router and assign /30 address directly to pfsense WAN and /25 addresses to LAN without success (the network was unstable).
For this reason we are in this situation now.
So, this being the case, I have to choose the second alternative solution (VIP + NAT).

Before closing this case, I would like to thank you very much for the time you dedicated to this case and the patience you have had.
I really appreciated. Many thanks for your support.