Strange behaviour for ICMP (ping) rule on WAN interface

[mauro.tridici]

@stephenw10 ok, I'm going to do it soon. How can I filter for ARP? I'm using the pfsense GUI. Do you think that I should do it using the command line?

[mauro.tridici]

@stephenw10

In attachment the pcap files.

em0 is the wan interface
em6.90 is the lan interface

thanks,
Mauro

em0_wan.pcap

em6.90_lan.pcap

[stephenw10]

Ok, so we can see the VM at .5 ARPing for the gateway at .1 and on both interfaces, both sides of the bridge.
But the gateway is not responding.

What is the upstream device at .1? It has a VMWare MAC address.

[mauro.tridici]

@stephenw10 many thanks for your patience and analysis.

Ok, so we can see the VM at .5 ARPing for the gateway at .1 and on both interfaces, both sides of the bridge.
But the gateway is not responding.

The upstream device at .1 is a VMware virtual machine that acts as virtual router (is a Zeroshell instance that has to be replaced very soon). It works as expected for the other pfsense interfaces, but it is not working for this kind of needs.

[stephenw10]

Can you check it's ARP table? Does it have an entry for .5?

Can you run a pcap on there and see if the ARP requests from .5 are arriving?

Either something is filtering them or it's just not or unable to respond.

Steve

[mauro.tridici]

@stephenw10

Can you check it's ARP table? Does it have an entry for .5?

I just checked the arp table, but I can't see an entry for .5 (please, take a look at file arp_table.txt

Can you run a pcap on there and see if the ARP requests from .5 are arriving?

I can create a pcap, but I captured the output of "tcpdump -I ETH02 arp" (please, take a look at file arp_requests.txt)

Either something is filtering them or it's just not or unable to respond.

I think there is no filter on .5 IP address.

arp_requests.txt

arp_table.txt

[stephenw10]

Hmm Ok. So we don't see the ARP requests from the VM at .5 arriving at the gateway. Even though they are on both interfaces in the bridge in pfSense.

However we do see the gateway ARPing for the .5 IP address and those packets never make it to pfSense.

pfSense is also in VMWare here?
It looks like something in the hypervisor filtering that traffic to me. The interfaces not in promiscuous mode maybe.

[mauro.tridici]

@stephenw10

Hmm Ok. So we don't see the ARP requests from the VM at .5 arriving at the gateway. Even though they are on both interfaces in the bridge in pfSense.
However we do see the gateway ARPing for the .5 IP address and those packets never make it to pfSense.

Yes, you are right.

pfSense is also in VMWare here?

Yes, virtual router and pfsense are on the same VMware hypervisor.
So, if the problem is that the shared virtual nic has to be in promiscuous mode, we can try to make a single change for both the service virtual machines.

It looks like something in the hypervisor filtering that traffic to me. The interfaces not in promiscuous mode maybe.

And I think that you are almost there because in the kernel log of the virtual router I see the following lines:

16:22:07 [9966861.866976] device ETH02 entered promiscuous mode
16:23:35 [9966950.185907] device ETH02 left promiscuous mode
18:48:06 [9975620.740132] device ETH02 entered promiscuous mode
18:49:13 [9975688.367374] device ETH02 left promiscuous mode
18:49:49 [9975724.468174] device ETH02 entered promiscuous mode
18:51:11 [9975806.502600] device ETH02 left promiscuous mode
18:52:07 [9975861.736268] device ETH02 entered promiscuous mode
18:56:19 [9976114.647032] device ETH02 left promiscuous mode

Ad the ETH02 is the involved interface (on the router side).
Do you think that I can simply try to enable the promiscuous mode on the virtual switch or should I do something also both on pfsense and router "operating system"?

[stephenw10]

It's more likely to be a settings on the pfSense WAN virtual NIC because that is trying to send traffic using a different MAC address and that's probably filtered.

Thought that doesn't explain why it doesn't see the broadcast ARP requests for .5 from the gateway. Perhaps those simply weren't happening during that capture.

Steve

[mauro.tridici]

Hello @stephenw10 ,

I'm back again :)

This is what I have done during the last hours.

On PFSENSE:

I noticed that, without enabling BRIDGE WAN<>LAN, there is no interface in promiscuous mode from the OS point of view.

[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: ifconfig|grep PROMISC
pflog0: flags=100<PROMISC> metric 0 mtu 33160

But, when I try to enable the BRIDGE WAN<>LAN, two interfaces come up configured in promiscuous mode:

[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: ifconfig | grep PROMISC
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
em6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
pflog0: flags=100<PROMISC> metric 0 mtu 33160
em6.90: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500

em0 -> WAN
em6 -> LAN
em6.90 -> VLAN on LAN

For this reason, I decided to enable the promiscuous mode also from the VMware hypervisor point of view.

on ROUTER:

Promiscuous mode has been enabled from OS and VMware point of view.

ifconfig ETH02 promisc
ifconfig ETH02:00 promisc

(ifconfig ETH02 -promisc to remove promiscous mode)

on VMWARE:

PROMISCOUS MODE AND ALLOW MAC CHANGE have been enabled on WAN and LAN vSwitch

on the VM:

Promiscuous mode has been enabled also on the interface of the VM

ifconfig ens192 promisc

BUT, unfortunately, I'm not able to ping the GW from the VM.
The behaviour is always the same.

Do you think that I have to do something else on the switches?

Thanks,
Mauro

[stephenw10]

Mmm, it has to be in promiscuous mode because it needs to use other MAC addresses in bridge mode. Because it's a VM that means the virtual NICs also need to and that's almost certainly what's happening.

Can you ping between the VM and pfSense? .2 and .5?

[mauro.tridici]

@stephenw10 thanks

Can you ping between the VM and pfSense? .2 and .5?

Sure, this is the result pinging VM from PFSENSE:

[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/var/log: ping y.y.y.5
PING y.y.y.5 (y.y.y.5): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
^Cping: sendto: Host is down

And this is the result pinging PFSENSE from VM:

test-vm: ping y.y.y.2
PING y.y.y.2 (y.y.y.2): 56 data bytes

[stephenw10]

Ok that result from pfSense implies the VM is not responding to ARP.

Do you see ARP requests in a pcap on em6.90?

[mauro.tridici]

@stephenw10

Yes, I can see a lot of ARP requests.
In particular, I noticed also some ARP requests related to .5

tcpdump -i em6.90 arp|grep y.y.y.5
00:59:34.641523 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:35.280550 ARP, Request who-has y.y.y.51 tell y.y.y.1, length 46
00:59:35.664545 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:36.688559 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:49.553602 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:50.576554 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:51.600577 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46

[stephenw10]

You don't have em6 in the bridge also do you? That would break the VLAN.

Bridging to VLANs has always been somewhat shaky!

Can you move that to a separate interface/vswitch? Or do the VLAN tagging in VMWare instead?

Steve

[stephenw10]

You could probably confirm that by running a pcap on the VM.

[mauro.tridici]

@stephenw10

You don't have em6 in the bridge also do you? That would break the VLAN.
Bridging to VLANs has always been somewhat shaky!

Yes, I confirm that only em6.90 is in the bridge :(
I'm sorry...

Can you move that to a separate interface/vswitch? Or do the VLAN tagging in VMWare instead?

I have to check if there is a free interface on the hypervisor, but, at this moment, I'm at home, not at the data center.
I have to think about it. I will let you know.

[mauro.tridici]

@stephenw10 running pcap on the VM I can see only the Rapid STP lines and the LLDP with the switch hostname. I think I can confirm that VLAN is broken as you said. (the VM interface is in promiscuous mode)

[stephenw10]

You should be able to bridge a VLAN to another interface like you are doing. What you definitely can't do it bridge the parent interface (em6) and then also use a VLAN on that. But when you through in virtual NICs you could be hitting some off loading bug that would not normally happen on em.

If you can use a discrete interface for 'Public LAN' and tag in VMWare instead I think there's a much better chance of making this work. Or perhaps you don't need VLANs at all if it's all virtual and in the same host.

Steve

[mauro.tridici]

You should be able to bridge a VLAN to another interface like you are doing. What you definitely can't do it bridge the parent interface (em6) and then also use a VLAN on that.

ok, fortunately I'm not doing it :)

But when you through in virtual NICs you could be hitting some off loading bug that would not normally happen on em.

ok, I am now understanding your doubt about the virtual NICs.

If you can use a discrete interface for 'Public LAN' and tag in VMWare instead I think there's a much better chance of making this work. Or perhaps you don't need VLANs at all if it's all virtual and in the same host.

I will return to work on Wednesday (tomorrow is a holiday) and I will check if there is a free interface left to devote to the "public LAN."
I don't know if I will be able to avoid using VLANs because all "public" traffic would travel on the switches' native lan (and that is not recommended).

Many thanks again for all the time you spent to help me. I really appreciated it.
See you soon.
Mauro