Strange behaviour for ICMP (ping) rule on WAN interface

mauro.tridici · Oct 31, 2022, 5:36 PM

@stephenw10 many thanks for your patience and analysis.

Ok, so we can see the VM at .5 ARPing for the gateway at .1 and on both interfaces, both sides of the bridge.
But the gateway is not responding.

The upstream device at .1 is a VMware virtual machine that acts as virtual router (is a Zeroshell instance that has to be replaced very soon). It works as expected for the other pfsense interfaces, but it is not working for this kind of needs.

stephenw10 · Oct 31, 2022, 5:43 PM

Can you check it's ARP table? Does it have an entry for .5?

Can you run a pcap on there and see if the ARP requests from .5 are arriving?

Either something is filtering them or it's just not or unable to respond.

Steve

mauro.tridici · Oct 31, 2022, 6:03 PM

@stephenw10

Can you check it's ARP table? Does it have an entry for .5?

I just checked the arp table, but I can't see an entry for .5 (please, take a look at file arp_table.txt

Can you run a pcap on there and see if the ARP requests from .5 are arriving?

I can create a pcap, but I captured the output of "tcpdump -I ETH02 arp" (please, take a look at file arp_requests.txt)

Either something is filtering them or it's just not or unable to respond.

I think there is no filter on .5 IP address.

arp_requests.txt

arp_table.txt

stephenw10 · Oct 31, 2022, 6:19 PM

Hmm Ok. So we don't see the ARP requests from the VM at .5 arriving at the gateway. Even though they are on both interfaces in the bridge in pfSense.

However we do see the gateway ARPing for the .5 IP address and those packets never make it to pfSense.

pfSense is also in VMWare here?
It looks like something in the hypervisor filtering that traffic to me. The interfaces not in promiscuous mode maybe.

mauro.tridici · Oct 31, 2022, 6:30 PM

@stephenw10

Hmm Ok. So we don't see the ARP requests from the VM at .5 arriving at the gateway. Even though they are on both interfaces in the bridge in pfSense.
However we do see the gateway ARPing for the .5 IP address and those packets never make it to pfSense.

Yes, you are right.

pfSense is also in VMWare here?

Yes, virtual router and pfsense are on the same VMware hypervisor.
So, if the problem is that the shared virtual nic has to be in promiscuous mode, we can try to make a single change for both the service virtual machines.

It looks like something in the hypervisor filtering that traffic to me. The interfaces not in promiscuous mode maybe.

And I think that you are almost there because in the kernel log of the virtual router I see the following lines:

16:22:07 [9966861.866976] device ETH02 entered promiscuous mode
16:23:35 [9966950.185907] device ETH02 left promiscuous mode
18:48:06 [9975620.740132] device ETH02 entered promiscuous mode
18:49:13 [9975688.367374] device ETH02 left promiscuous mode
18:49:49 [9975724.468174] device ETH02 entered promiscuous mode
18:51:11 [9975806.502600] device ETH02 left promiscuous mode
18:52:07 [9975861.736268] device ETH02 entered promiscuous mode
18:56:19 [9976114.647032] device ETH02 left promiscuous mode

Ad the ETH02 is the involved interface (on the router side).
Do you think that I can simply try to enable the promiscuous mode on the virtual switch or should I do something also both on pfsense and router "operating system"?

stephenw10 · Oct 31, 2022, 6:45 PM

It's more likely to be a settings on the pfSense WAN virtual NIC because that is trying to send traffic using a different MAC address and that's probably filtered.

Thought that doesn't explain why it doesn't see the broadcast ARP requests for .5 from the gateway. Perhaps those simply weren't happening during that capture.

Steve

mauro.tridici · Oct 31, 2022, 10:42 PM

Hello @stephenw10 ,

I'm back again :)

This is what I have done during the last hours.

On PFSENSE:

I noticed that, without enabling BRIDGE WAN<>LAN, there is no interface in promiscuous mode from the OS point of view.

[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: ifconfig|grep PROMISC
pflog0: flags=100<PROMISC> metric 0 mtu 33160

But, when I try to enable the BRIDGE WAN<>LAN, two interfaces come up configured in promiscuous mode:

[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: ifconfig | grep PROMISC
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
em6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
pflog0: flags=100<PROMISC> metric 0 mtu 33160
em6.90: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500

em0 -> WAN
em6 -> LAN
em6.90 -> VLAN on LAN

For this reason, I decided to enable the promiscuous mode also from the VMware hypervisor point of view.

on ROUTER:

Promiscuous mode has been enabled from OS and VMware point of view.

ifconfig ETH02 promisc
ifconfig ETH02:00 promisc

(ifconfig ETH02 -promisc to remove promiscous mode)

on VMWARE:

PROMISCOUS MODE AND ALLOW MAC CHANGE have been enabled on WAN and LAN vSwitch

on the VM:

Promiscuous mode has been enabled also on the interface of the VM

ifconfig ens192 promisc

BUT, unfortunately, I'm not able to ping the GW from the VM.
The behaviour is always the same.

Do you think that I have to do something else on the switches?

Thanks,
Mauro

stephenw10 · Oct 31, 2022, 11:30 PM

Mmm, it has to be in promiscuous mode because it needs to use other MAC addresses in bridge mode. Because it's a VM that means the virtual NICs also need to and that's almost certainly what's happening.

Can you ping between the VM and pfSense? .2 and .5?

mauro.tridici · Oct 31, 2022, 11:40 PM

@stephenw10 thanks

Can you ping between the VM and pfSense? .2 and .5?

Sure, this is the result pinging VM from PFSENSE:

[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/var/log: ping y.y.y.5
PING y.y.y.5 (y.y.y.5): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
^Cping: sendto: Host is down

And this is the result pinging PFSENSE from VM:

test-vm: ping y.y.y.2
PING y.y.y.2 (y.y.y.2): 56 data bytes

stephenw10 · Nov 1, 2022, 12:04 AM

Ok that result from pfSense implies the VM is not responding to ARP.

Do you see ARP requests in a pcap on em6.90?

mauro.tridici · Nov 1, 2022, 12:04 AM

@stephenw10

Yes, I can see a lot of ARP requests.
In particular, I noticed also some ARP requests related to .5

tcpdump -i em6.90 arp|grep y.y.y.5
00:59:34.641523 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:35.280550 ARP, Request who-has y.y.y.51 tell y.y.y.1, length 46
00:59:35.664545 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:36.688559 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:49.553602 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:50.576554 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:51.600577 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46

stephenw10 · Nov 1, 2022, 12:19 AM

You don't have em6 in the bridge also do you? That would break the VLAN.

Bridging to VLANs has always been somewhat shaky!

Can you move that to a separate interface/vswitch? Or do the VLAN tagging in VMWare instead?

Steve

stephenw10 · Nov 1, 2022, 12:20 AM

You could probably confirm that by running a pcap on the VM.

mauro.tridici · Nov 1, 2022, 12:29 AM

@stephenw10

You don't have em6 in the bridge also do you? That would break the VLAN.
Bridging to VLANs has always been somewhat shaky!

Yes, I confirm that only em6.90 is in the bridge :(
I'm sorry...

Can you move that to a separate interface/vswitch? Or do the VLAN tagging in VMWare instead?

I have to check if there is a free interface on the hypervisor, but, at this moment, I'm at home, not at the data center.
I have to think about it. I will let you know.

mauro.tridici · Nov 1, 2022, 12:32 AM

@stephenw10 running pcap on the VM I can see only the Rapid STP lines and the LLDP with the switch hostname. I think I can confirm that VLAN is broken as you said. (the VM interface is in promiscuous mode)

stephenw10 · Nov 1, 2022, 1:13 AM

You should be able to bridge a VLAN to another interface like you are doing. What you definitely can't do it bridge the parent interface (em6) and then also use a VLAN on that. But when you through in virtual NICs you could be hitting some off loading bug that would not normally happen on em.

If you can use a discrete interface for 'Public LAN' and tag in VMWare instead I think there's a much better chance of making this work. Or perhaps you don't need VLANs at all if it's all virtual and in the same host.

Steve

mauro.tridici · Nov 1, 2022, 1:13 AM

You should be able to bridge a VLAN to another interface like you are doing. What you definitely can't do it bridge the parent interface (em6) and then also use a VLAN on that.

ok, fortunately I'm not doing it :)

But when you through in virtual NICs you could be hitting some off loading bug that would not normally happen on em.

ok, I am now understanding your doubt about the virtual NICs.

If you can use a discrete interface for 'Public LAN' and tag in VMWare instead I think there's a much better chance of making this work. Or perhaps you don't need VLANs at all if it's all virtual and in the same host.

I will return to work on Wednesday (tomorrow is a holiday) and I will check if there is a free interface left to devote to the "public LAN."
I don't know if I will be able to avoid using VLANs because all "public" traffic would travel on the switches' native lan (and that is not recommended).

Many thanks again for all the time you spent to help me. I really appreciated it.
See you soon.
Mauro

stephenw10 · Nov 1, 2022, 1:25 AM

Ah, is it not just a vswitch in VMWare between pfSense and the .5 VM then?

Even so you can probably just do the tagging in VMWare so pfSense just sees two separate NICs.

mauro.tridici · Nov 2, 2022, 8:31 AM

@stephenw10 Good morning, Stephen.

This is to inform you that I just checked the number of free interfaces on the VMware hypervisor. Unfortunately, there is no free interface to be dedicated to the "public LAN".

Anyway, before giving up, I made a search on pfSense Forum looking for "pfsense vlan bridge" and I found this interesting topic:

https://forum.netgate.com/topic/136900/help-with-vlans-in-bridge/23?_=1667375382000

In particular, at the end of discussion, I read that "broonu" user says:

With tcpdump i see the ARP Request but pfsense dont send the ARP Reply.
Im going to clear everything and reconfigure from scratch.

and it seems he solved his issue with the following action:

@delerict thank you for your time and help!
it was a vmware misconfiguration, e1000 nic instead of vmxnet3.

After reading that, I checked the type of virtual nic assigned to WAN and "public LAN" interfaces of my pfSense instance and I noticed that the interfaces type is E1000. So, I decided to reply to "broonu" user with the following message:

@broonu Hello, sorry if I'm replying to this old topic, but I'm experiencing the same problem trying to bridge the WAN interface with a VLAN created on a LAN interface.

The behavior is almost the same: no reply to ARP requests from pfsense + I cant ping the pfsense upstream gateway.

Before giving up, I noticed that the WAN and LAN interfaces are E1000 (not VMXNET3).
I would like to change the nic type as last attempt.
Anyway, before doing that, I would like to know if there is a particolar relation between bridge and vmxnet3.

Could you please help me?
Thanks

In addition, I would like to answer your question:

Ah, is it not just a vswitch in VMWare between pfSense and the .5 VM then?

Yes, it is not just a vswitch in VMware between pfSense and the .5 VM. Virtual router and pfsense are on the same hypervisor, but the "client" machines (virtual and physical ones) are connected to pfSense through a switch.

Even so you can probably just do the tagging in VMWare so pfSense just sees two separate NICs.

This is the current configuration:

physical pfSense interface em0 (WAN) is connected to the router;
physical pfSense interface em6 is connected to a vswitch (named "LAN") that has been configured to manage/accept all VLANs (trunk mode);
on vmware side, the vswitch mentioned above is logically connected to a TRUNK port on the physical switch;
on pfsense side, several VLANs have been created adding tags (for example, "VLAN em6.90", "VLAN em6.10" and so on);
the VM is connected to an ACCESS port (with VLAN ID 90) on the same physical switch.

If you think that the main cause is still the VLAN over the bridge (and not the E1000 nic type), could you help me to better understand how to apply your last suggestion?

Thanks in advance,
Mauro

mauro.tridici · Nov 2, 2022, 9:26 AM

@stephenw10

I think I understand your suggestion:

On vmware side, on top fo the vswitch "LAN" (the one that accepts every VLAN), I created a portgroup named "Public LAN" with tag 90.
I also added a virtual nic to the pfsense instance (let's say, em8) and I connected this nic to the "Public LAN" portgroup.

So, in this new situation, em8 will be a new interface of pfsense without any VLAN on top of itself.

As soon as possible, I will do a reboot of pfsense and I will test the bridge again.
If it will fail, I will try also to change E1000 to VMXNET3.

Crossing fingers,
Mauro