Strange behaviour for ICMP (ping) rule on WAN interface
-
Hmm, do you see an ARP entry for the gateway on the VM?
How can check that?
I executed the "arp -a" command on the VM and the result is:
gateway (y.y.y.1) at <incomplete> on ens192The subnet mask seems to be ok.
-
Ok, so it isn't seeing ARP replies from the gateway. Are there any complete ARP entries there beyond it's own IP? Presumably it has the pfSense y.y.y.2 IP because it does send ICMP packets to that.
The next thing I would do then is run a pcap on the pfSense WAN and see traffic is there. It should have the ARP requests from the VM.
Steve
-
Ok, so it isn't seeing ARP replies from the gateway. Are there any complete ARP entries there beyond it's own IP? Presumably it has the pfSense y.y.y.2 IP because it does send ICMP packets to that.
Mmmh, unfortunately no. After changing the VM gateway from y.y.y.2 to y.y.y.1, no complete ARP entries are listed.
Using the old (but wrong) configuration (with y.y.y.2 as gateway for the VM), I can see the complete ARP entry for y.y.y.2 IP.The next thing I would do then is run a pcap on the pfSense WAN and see traffic is there. It should have the ARP requests from the VM.
Let's try! I executed the "ping y.y.y.1" from the VM and I captured the pcap file on WAN interface, IPv4, Any protocol, host address y.y.y.0/25 (at the end, I revealed the subnet...). If I'm not wrong the .5 IP is not listed in pcap. I will send you the subnet details in a private message.
-
-
That pcap spans only 2.2 seconds. You will probably need to filter it by something. Here what we are looking for is ARP requests from .5 for .1 so I'd filter for just ARP in initially.
Also run the pcap on the Public LAN interface too to make sure they are arrivign there.The bridge should just pass those.
Also you should still be able to try to ping the .2 address and see the ARP table populated for that.
Steve
-
@stephenw10 ok, I'm going to do it soon. How can I filter for ARP? I'm using the pfsense GUI. Do you think that I should do it using the command line?
-
In attachment the pcap files.
em0 is the wan interface
em6.90 is the lan interfacethanks,
Mauro -
Ok, so we can see the VM at .5 ARPing for the gateway at .1 and on both interfaces, both sides of the bridge.
But the gateway is not responding.What is the upstream device at .1? It has a VMWare MAC address.
-
@stephenw10 many thanks for your patience and analysis.
Ok, so we can see the VM at .5 ARPing for the gateway at .1 and on both interfaces, both sides of the bridge.
But the gateway is not responding.The upstream device at .1 is a VMware virtual machine that acts as virtual router (is a Zeroshell instance that has to be replaced very soon). It works as expected for the other pfsense interfaces, but it is not working for this kind of needs.
-
Can you check it's ARP table? Does it have an entry for .5?
Can you run a pcap on there and see if the ARP requests from .5 are arriving?
Either something is filtering them or it's just not or unable to respond.
Steve
-
Can you check it's ARP table? Does it have an entry for .5?
I just checked the arp table, but I can't see an entry for .5 (please, take a look at file arp_table.txt
Can you run a pcap on there and see if the ARP requests from .5 are arriving?
I can create a pcap, but I captured the output of "tcpdump -I ETH02 arp" (please, take a look at file arp_requests.txt)
Either something is filtering them or it's just not or unable to respond.
I think there is no filter on .5 IP address.
-
Hmm Ok. So we don't see the ARP requests from the VM at .5 arriving at the gateway. Even though they are on both interfaces in the bridge in pfSense.
However we do see the gateway ARPing for the .5 IP address and those packets never make it to pfSense.
pfSense is also in VMWare here?
It looks like something in the hypervisor filtering that traffic to me. The interfaces not in promiscuous mode maybe. -
Hmm Ok. So we don't see the ARP requests from the VM at .5 arriving at the gateway. Even though they are on both interfaces in the bridge in pfSense.
However we do see the gateway ARPing for the .5 IP address and those packets never make it to pfSense.Yes, you are right.
pfSense is also in VMWare here?
Yes, virtual router and pfsense are on the same VMware hypervisor.
So, if the problem is that the shared virtual nic has to be in promiscuous mode, we can try to make a single change for both the service virtual machines.It looks like something in the hypervisor filtering that traffic to me. The interfaces not in promiscuous mode maybe.
And I think that you are almost there because in the kernel log of the virtual router I see the following lines:
16:22:07 [9966861.866976] device ETH02 entered promiscuous mode
16:23:35 [9966950.185907] device ETH02 left promiscuous mode
18:48:06 [9975620.740132] device ETH02 entered promiscuous mode
18:49:13 [9975688.367374] device ETH02 left promiscuous mode
18:49:49 [9975724.468174] device ETH02 entered promiscuous mode
18:51:11 [9975806.502600] device ETH02 left promiscuous mode
18:52:07 [9975861.736268] device ETH02 entered promiscuous mode
18:56:19 [9976114.647032] device ETH02 left promiscuous modeAd the ETH02 is the involved interface (on the router side).
Do you think that I can simply try to enable the promiscuous mode on the virtual switch or should I do something also both on pfsense and router "operating system"? -
It's more likely to be a settings on the pfSense WAN virtual NIC because that is trying to send traffic using a different MAC address and that's probably filtered.
Thought that doesn't explain why it doesn't see the broadcast ARP requests for .5 from the gateway. Perhaps those simply weren't happening during that capture.
Steve
-
Hello @stephenw10 ,
I'm back again :)
This is what I have done during the last hours.
On PFSENSE:
I noticed that, without enabling BRIDGE WAN<>LAN, there is no interface in promiscuous mode from the OS point of view.
[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: ifconfig|grep PROMISC
pflog0: flags=100<PROMISC> metric 0 mtu 33160But, when I try to enable the BRIDGE WAN<>LAN, two interfaces come up configured in promiscuous mode:
[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: ifconfig | grep PROMISC
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
em6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
pflog0: flags=100<PROMISC> metric 0 mtu 33160
em6.90: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500em0 -> WAN
em6 -> LAN
em6.90 -> VLAN on LANFor this reason, I decided to enable the promiscuous mode also from the VMware hypervisor point of view.
on ROUTER:
Promiscuous mode has been enabled from OS and VMware point of view.
ifconfig ETH02 promisc
ifconfig ETH02:00 promisc(ifconfig ETH02 -promisc to remove promiscous mode)
on VMWARE:
PROMISCOUS MODE AND ALLOW MAC CHANGE have been enabled on WAN and LAN vSwitch
on the VM:
Promiscuous mode has been enabled also on the interface of the VM
ifconfig ens192 promisc
BUT, unfortunately, I'm not able to ping the GW from the VM.
The behaviour is always the same.Do you think that I have to do something else on the switches?
Thanks,
Mauro -
Mmm, it has to be in promiscuous mode because it needs to use other MAC addresses in bridge mode. Because it's a VM that means the virtual NICs also need to and that's almost certainly what's happening.
Can you ping between the VM and pfSense? .2 and .5?
-
@stephenw10 thanks
Can you ping between the VM and pfSense? .2 and .5?
Sure, this is the result pinging VM from PFSENSE:
[2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/var/log: ping y.y.y.5
PING y.y.y.5 (y.y.y.5): 56 data bytes
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
ping: sendto: Host is down
^Cping: sendto: Host is downAnd this is the result pinging PFSENSE from VM:
test-vm: ping y.y.y.2
PING y.y.y.2 (y.y.y.2): 56 data bytes -
Ok that result from pfSense implies the VM is not responding to ARP.
Do you see ARP requests in a pcap on em6.90?
-
Yes, I can see a lot of ARP requests.
In particular, I noticed also some ARP requests related to .5tcpdump -i em6.90 arp|grep y.y.y.5
00:59:34.641523 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:35.280550 ARP, Request who-has y.y.y.51 tell y.y.y.1, length 46
00:59:35.664545 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:36.688559 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:49.553602 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:50.576554 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
00:59:51.600577 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46 -
You don't have em6 in the bridge also do you? That would break the VLAN.
Bridging to VLANs has always been somewhat shaky!
Can you move that to a separate interface/vswitch? Or do the VLAN tagging in VMWare instead?
Steve
