Strange behaviour for ICMP (ping) rule on WAN interface

mauro.tridici · Oct 31, 2022, 3:44 PM

Hmm, do you see an ARP entry for the gateway on the VM?

How can check that?
I executed the "arp -a" command on the VM and the result is:
gateway (y.y.y.1) at <incomplete> on ens192

The subnet mask seems to be ok.

stephenw10 · Oct 31, 2022, 4:01 PM

Ok, so it isn't seeing ARP replies from the gateway. Are there any complete ARP entries there beyond it's own IP? Presumably it has the pfSense y.y.y.2 IP because it does send ICMP packets to that.

The next thing I would do then is run a pcap on the pfSense WAN and see traffic is there. It should have the ARP requests from the VM.

Steve

mauro.tridici · Oct 31, 2022, 4:22 PM

@stephenw10

Ok, so it isn't seeing ARP replies from the gateway. Are there any complete ARP entries there beyond it's own IP? Presumably it has the pfSense y.y.y.2 IP because it does send ICMP packets to that.

Mmmh, unfortunately no. After changing the VM gateway from y.y.y.2 to y.y.y.1, no complete ARP entries are listed.
Using the old (but wrong) configuration (with y.y.y.2 as gateway for the VM), I can see the complete ARP entry for y.y.y.2 IP.

The next thing I would do then is run a pcap on the pfSense WAN and see traffic is there. It should have the ARP requests from the VM.

Let's try! I executed the "ping y.y.y.1" from the VM and I captured the pcap file on WAN interface, IPv4, Any protocol, host address y.y.y.0/25 (at the end, I revealed the subnet...). If I'm not wrong the .5 IP is not listed in pcap. I will send you the subnet details in a private message.

mauro.tridici · Oct 31, 2022, 4:27 PM

@stephenw10 packetcapture.cap

stephenw10 · Oct 31, 2022, 4:31 PM

That pcap spans only 2.2 seconds. You will probably need to filter it by something. Here what we are looking for is ARP requests from .5 for .1 so I'd filter for just ARP in initially.
Also run the pcap on the Public LAN interface too to make sure they are arrivign there.

The bridge should just pass those.

Also you should still be able to try to ping the .2 address and see the ARP table populated for that.

Steve

mauro.tridici · Oct 31, 2022, 4:34 PM

@stephenw10 ok, I'm going to do it soon. How can I filter for ARP? I'm using the pfsense GUI. Do you think that I should do it using the command line?

mauro.tridici · Oct 31, 2022, 4:42 PM

@stephenw10

In attachment the pcap files.

em0 is the wan interface
em6.90 is the lan interface

thanks,
Mauro

em0_wan.pcap

em6.90_lan.pcap

stephenw10 · Oct 31, 2022, 5:29 PM

Ok, so we can see the VM at .5 ARPing for the gateway at .1 and on both interfaces, both sides of the bridge.
But the gateway is not responding.

What is the upstream device at .1? It has a VMWare MAC address.

mauro.tridici · Oct 31, 2022, 5:36 PM

@stephenw10 many thanks for your patience and analysis.

Ok, so we can see the VM at .5 ARPing for the gateway at .1 and on both interfaces, both sides of the bridge.
But the gateway is not responding.

The upstream device at .1 is a VMware virtual machine that acts as virtual router (is a Zeroshell instance that has to be replaced very soon). It works as expected for the other pfsense interfaces, but it is not working for this kind of needs.

stephenw10 · Oct 31, 2022, 5:43 PM

Can you check it's ARP table? Does it have an entry for .5?

Can you run a pcap on there and see if the ARP requests from .5 are arriving?

Either something is filtering them or it's just not or unable to respond.

Steve

mauro.tridici · Oct 31, 2022, 6:03 PM

@stephenw10

Can you check it's ARP table? Does it have an entry for .5?

I just checked the arp table, but I can't see an entry for .5 (please, take a look at file arp_table.txt

Can you run a pcap on there and see if the ARP requests from .5 are arriving?

I can create a pcap, but I captured the output of "tcpdump -I ETH02 arp" (please, take a look at file arp_requests.txt)

Either something is filtering them or it's just not or unable to respond.

I think there is no filter on .5 IP address.

arp_requests.txt

arp_table.txt

stephenw10 · Oct 31, 2022, 6:19 PM

Hmm Ok. So we don't see the ARP requests from the VM at .5 arriving at the gateway. Even though they are on both interfaces in the bridge in pfSense.

However we do see the gateway ARPing for the .5 IP address and those packets never make it to pfSense.

pfSense is also in VMWare here?
It looks like something in the hypervisor filtering that traffic to me. The interfaces not in promiscuous mode maybe.

mauro.tridici · Oct 31, 2022, 6:30 PM

@stephenw10

Hmm Ok. So we don't see the ARP requests from the VM at .5 arriving at the gateway. Even though they are on both interfaces in the bridge in pfSense.
However we do see the gateway ARPing for the .5 IP address and those packets never make it to pfSense.

Yes, you are right.

pfSense is also in VMWare here?

Yes, virtual router and pfsense are on the same VMware hypervisor.
So, if the problem is that the shared virtual nic has to be in promiscuous mode, we can try to make a single change for both the service virtual machines.

It looks like something in the hypervisor filtering that traffic to me. The interfaces not in promiscuous mode maybe.

And I think that you are almost there because in the kernel log of the virtual router I see the following lines:

16:22:07 [9966861.866976] device ETH02 entered promiscuous mode
16:23:35 [9966950.185907] device ETH02 left promiscuous mode
18:48:06 [9975620.740132] device ETH02 entered promiscuous mode
18:49:13 [9975688.367374] device ETH02 left promiscuous mode
18:49:49 [9975724.468174] device ETH02 entered promiscuous mode
18:51:11 [9975806.502600] device ETH02 left promiscuous mode
18:52:07 [9975861.736268] device ETH02 entered promiscuous mode
18:56:19 [9976114.647032] device ETH02 left promiscuous mode

Ad the ETH02 is the involved interface (on the router side).
Do you think that I can simply try to enable the promiscuous mode on the virtual switch or should I do something also both on pfsense and router "operating system"?

stephenw10 · Oct 31, 2022, 6:45 PM

It's more likely to be a settings on the pfSense WAN virtual NIC because that is trying to send traffic using a different MAC address and that's probably filtered.

Thought that doesn't explain why it doesn't see the broadcast ARP requests for .5 from the gateway. Perhaps those simply weren't happening during that capture.

Steve

mauro.tridici · Oct 31, 2022, 10:42 PM

Hello @stephenw10 ,

I'm back again :)

This is what I have done during the last hours.

On PFSENSE:

I noticed that, without enabling BRIDGE WAN<>LAN, there is no interface in promiscuous mode from the OS point of view.

[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: ifconfig|grep PROMISC
pflog0: flags=100<PROMISC> metric 0 mtu 33160

But, when I try to enable the BRIDGE WAN<>LAN, two interfaces come up configured in promiscuous mode:

[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: ifconfig | grep PROMISC
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
em6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
pflog0: flags=100<PROMISC> metric 0 mtu 33160
em6.90: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500

em0 -> WAN
em6 -> LAN
em6.90 -> VLAN on LAN

For this reason, I decided to enable the promiscuous mode also from the VMware hypervisor point of view.

on ROUTER:

Promiscuous mode has been enabled from OS and VMware point of view.

ifconfig ETH02 promisc
ifconfig ETH02:00 promisc

(ifconfig ETH02 -promisc to remove promiscous mode)

on VMWARE:

PROMISCOUS MODE AND ALLOW MAC CHANGE have been enabled on WAN and LAN vSwitch

on the VM:

Promiscuous mode has been enabled also on the interface of the VM

ifconfig ens192 promisc

BUT, unfortunately, I'm not able to ping the GW from the VM.
The behaviour is always the same.

Do you think that I have to do something else on the switches?

Thanks,
Mauro

stephenw10 · Oct 31, 2022, 11:30 PM

Mmm, it has to be in promiscuous mode because it needs to use other MAC addresses in bridge mode. Because it's a VM that means the virtual NICs also need to and that's almost certainly what's happening.

Can you ping between the VM and pfSense? .2 and .5?

mauro.tridici · Oct 31, 2022, 11:40 PM

@stephenw10 thanks

Can you ping between the VM and pfSense? .2 and .5?

Sure, this is the result pinging VM from PFSENSE:

[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/var/log: ping y.y.y.5
PING y.y.y.5 (y.y.y.5): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
^Cping: sendto: Host is down

And this is the result pinging PFSENSE from VM:

test-vm: ping y.y.y.2
PING y.y.y.2 (y.y.y.2): 56 data bytes

stephenw10 · Nov 1, 2022, 12:04 AM

Ok that result from pfSense implies the VM is not responding to ARP.

Do you see ARP requests in a pcap on em6.90?

mauro.tridici · Nov 1, 2022, 12:04 AM

@stephenw10

Yes, I can see a lot of ARP requests.
In particular, I noticed also some ARP requests related to .5

tcpdump -i em6.90 arp|grep y.y.y.5
00:59:34.641523 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:35.280550 ARP, Request who-has y.y.y.51 tell y.y.y.1, length 46
00:59:35.664545 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:36.688559 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:49.553602 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:50.576554 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:51.600577 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46

stephenw10 · Nov 1, 2022, 12:19 AM

You don't have em6 in the bridge also do you? That would break the VLAN.

Bridging to VLANs has always been somewhat shaky!

Can you move that to a separate interface/vswitch? Or do the VLAN tagging in VMWare instead?

Steve