• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Strange behaviour for ICMP (ping) rule on WAN interface

Scheduled Pinned Locked Moved General pfSense Questions
92 Posts 3 Posters 16.7k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • M
    mauro.tridici @stephenw10
    last edited by Oct 31, 2022, 4:27 PM

    @stephenw10 packetcapture.cap

    1 Reply Last reply Reply Quote 0
    • S
      stephenw10 Netgate Administrator
      last edited by stephenw10 Oct 31, 2022, 4:31 PM Oct 31, 2022, 4:30 PM

      That pcap spans only 2.2 seconds. You will probably need to filter it by something. Here what we are looking for is ARP requests from .5 for .1 so I'd filter for just ARP in initially.
      Also run the pcap on the Public LAN interface too to make sure they are arrivign there.

      The bridge should just pass those.

      Also you should still be able to try to ping the .2 address and see the ARP table populated for that.

      Steve

      M 2 Replies Last reply Oct 31, 2022, 4:34 PM Reply Quote 1
      • M
        mauro.tridici @stephenw10
        last edited by Oct 31, 2022, 4:34 PM

        @stephenw10 ok, I'm going to do it soon. How can I filter for ARP? I'm using the pfsense GUI. Do you think that I should do it using the command line?

        1 Reply Last reply Reply Quote 0
        • M
          mauro.tridici @stephenw10
          last edited by Oct 31, 2022, 4:42 PM

          @stephenw10

          In attachment the pcap files.

          em0 is the wan interface
          em6.90 is the lan interface

          thanks,
          Mauro

          em0_wan.pcap

          em6.90_lan.pcap

          1 Reply Last reply Reply Quote 0
          • S
            stephenw10 Netgate Administrator
            last edited by Oct 31, 2022, 5:29 PM

            Ok, so we can see the VM at .5 ARPing for the gateway at .1 and on both interfaces, both sides of the bridge.
            But the gateway is not responding.

            What is the upstream device at .1? It has a VMWare MAC address.

            M 1 Reply Last reply Oct 31, 2022, 5:36 PM Reply Quote 1
            • M
              mauro.tridici @stephenw10
              last edited by Oct 31, 2022, 5:36 PM

              @stephenw10 many thanks for your patience and analysis.

              Ok, so we can see the VM at .5 ARPing for the gateway at .1 and on both interfaces, both sides of the bridge.
              But the gateway is not responding.

              The upstream device at .1 is a VMware virtual machine that acts as virtual router (is a Zeroshell instance that has to be replaced very soon). It works as expected for the other pfsense interfaces, but it is not working for this kind of needs.

              1 Reply Last reply Reply Quote 0
              • S
                stephenw10 Netgate Administrator
                last edited by Oct 31, 2022, 5:43 PM

                Can you check it's ARP table? Does it have an entry for .5?

                Can you run a pcap on there and see if the ARP requests from .5 are arriving?

                Either something is filtering them or it's just not or unable to respond.

                Steve

                M 1 Reply Last reply Oct 31, 2022, 6:03 PM Reply Quote 0
                • M
                  mauro.tridici @stephenw10
                  last edited by Oct 31, 2022, 6:03 PM

                  @stephenw10

                  Can you check it's ARP table? Does it have an entry for .5?

                  I just checked the arp table, but I can't see an entry for .5 (please, take a look at file arp_table.txt

                  Can you run a pcap on there and see if the ARP requests from .5 are arriving?

                  I can create a pcap, but I captured the output of "tcpdump -I ETH02 arp" (please, take a look at file arp_requests.txt)

                  Either something is filtering them or it's just not or unable to respond.

                  I think there is no filter on .5 IP address.

                  arp_requests.txt

                  arp_table.txt

                  1 Reply Last reply Reply Quote 0
                  • S
                    stephenw10 Netgate Administrator
                    last edited by Oct 31, 2022, 6:19 PM

                    Hmm Ok. So we don't see the ARP requests from the VM at .5 arriving at the gateway. Even though they are on both interfaces in the bridge in pfSense.

                    However we do see the gateway ARPing for the .5 IP address and those packets never make it to pfSense.

                    pfSense is also in VMWare here?
                    It looks like something in the hypervisor filtering that traffic to me. The interfaces not in promiscuous mode maybe.

                    M 1 Reply Last reply Oct 31, 2022, 6:30 PM Reply Quote 0
                    • M
                      mauro.tridici @stephenw10
                      last edited by Oct 31, 2022, 6:30 PM

                      @stephenw10

                      Hmm Ok. So we don't see the ARP requests from the VM at .5 arriving at the gateway. Even though they are on both interfaces in the bridge in pfSense.
                      However we do see the gateway ARPing for the .5 IP address and those packets never make it to pfSense.

                      Yes, you are right.

                      pfSense is also in VMWare here?

                      Yes, virtual router and pfsense are on the same VMware hypervisor.
                      So, if the problem is that the shared virtual nic has to be in promiscuous mode, we can try to make a single change for both the service virtual machines.

                      It looks like something in the hypervisor filtering that traffic to me. The interfaces not in promiscuous mode maybe.

                      And I think that you are almost there because in the kernel log of the virtual router I see the following lines:

                      16:22:07 [9966861.866976] device ETH02 entered promiscuous mode
                      16:23:35 [9966950.185907] device ETH02 left promiscuous mode
                      18:48:06 [9975620.740132] device ETH02 entered promiscuous mode
                      18:49:13 [9975688.367374] device ETH02 left promiscuous mode
                      18:49:49 [9975724.468174] device ETH02 entered promiscuous mode
                      18:51:11 [9975806.502600] device ETH02 left promiscuous mode
                      18:52:07 [9975861.736268] device ETH02 entered promiscuous mode
                      18:56:19 [9976114.647032] device ETH02 left promiscuous mode

                      Ad the ETH02 is the involved interface (on the router side).
                      Do you think that I can simply try to enable the promiscuous mode on the virtual switch or should I do something also both on pfsense and router "operating system"?

                      1 Reply Last reply Reply Quote 0
                      • S
                        stephenw10 Netgate Administrator
                        last edited by Oct 31, 2022, 6:45 PM

                        It's more likely to be a settings on the pfSense WAN virtual NIC because that is trying to send traffic using a different MAC address and that's probably filtered.

                        Thought that doesn't explain why it doesn't see the broadcast ARP requests for .5 from the gateway. Perhaps those simply weren't happening during that capture.

                        Steve

                        M 1 Reply Last reply Oct 31, 2022, 10:42 PM Reply Quote 0
                        • M
                          mauro.tridici @stephenw10
                          last edited by Oct 31, 2022, 10:42 PM

                          Hello @stephenw10 ,

                          I'm back again :)

                          This is what I have done during the last hours.

                          On PFSENSE:

                          I noticed that, without enabling BRIDGE WAN<>LAN, there is no interface in promiscuous mode from the OS point of view.

                          [2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: ifconfig|grep PROMISC
                          pflog0: flags=100<PROMISC> metric 0 mtu 33160

                          But, when I try to enable the BRIDGE WAN<>LAN, two interfaces come up configured in promiscuous mode:

                          [2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/root: ifconfig | grep PROMISC
                          em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
                          em6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
                          pflog0: flags=100<PROMISC> metric 0 mtu 33160
                          em6.90: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500

                          em0 -> WAN
                          em6 -> LAN
                          em6.90 -> VLAN on LAN

                          For this reason, I decided to enable the promiscuous mode also from the VMware hypervisor point of view.

                          on ROUTER:

                          Promiscuous mode has been enabled from OS and VMware point of view.

                          ifconfig ETH02 promisc
                          ifconfig ETH02:00 promisc

                          (ifconfig ETH02 -promisc to remove promiscous mode)

                          on VMWARE:

                          PROMISCOUS MODE AND ALLOW MAC CHANGE have been enabled on WAN and LAN vSwitch

                          on the VM:

                          Promiscuous mode has been enabled also on the interface of the VM

                          ifconfig ens192 promisc

                          BUT, unfortunately, I'm not able to ping the GW from the VM.
                          The behaviour is always the same.

                          Do you think that I have to do something else on the switches?

                          Thanks,
                          Mauro

                          1 Reply Last reply Reply Quote 0
                          • S
                            stephenw10 Netgate Administrator
                            last edited by Oct 31, 2022, 11:30 PM

                            Mmm, it has to be in promiscuous mode because it needs to use other MAC addresses in bridge mode. Because it's a VM that means the virtual NICs also need to and that's almost certainly what's happening.

                            Can you ping between the VM and pfSense? .2 and .5?

                            M 1 Reply Last reply Oct 31, 2022, 11:40 PM Reply Quote 0
                            • M
                              mauro.tridici @stephenw10
                              last edited by Oct 31, 2022, 11:40 PM

                              @stephenw10 thanks

                              Can you ping between the VM and pfSense? .2 and .5?

                              Sure, this is the result pinging VM from PFSENSE:

                              [2.5.2-RELEASE][admin@pfSense_LAN_CMCC.home.arpa]/var/log: ping y.y.y.5
                              PING y.y.y.5 (y.y.y.5): 56 data bytes
                              ping: sendto: Host is down
                              ping: sendto: Host is down
                              ping: sendto: Host is down
                              ping: sendto: Host is down
                              ping: sendto: Host is down
                              ping: sendto: Host is down
                              ^Cping: sendto: Host is down

                              And this is the result pinging PFSENSE from VM:

                              test-vm: ping y.y.y.2
                              PING y.y.y.2 (y.y.y.2): 56 data bytes

                              1 Reply Last reply Reply Quote 0
                              • S
                                stephenw10 Netgate Administrator
                                last edited by Oct 31, 2022, 11:57 PM

                                Ok that result from pfSense implies the VM is not responding to ARP.

                                Do you see ARP requests in a pcap on em6.90?

                                M 1 Reply Last reply Nov 1, 2022, 12:04 AM Reply Quote 0
                                • M
                                  mauro.tridici @stephenw10
                                  last edited by Nov 1, 2022, 12:04 AM

                                  @stephenw10

                                  Yes, I can see a lot of ARP requests.
                                  In particular, I noticed also some ARP requests related to .5

                                  tcpdump -i em6.90 arp|grep y.y.y.5
                                  00:59:34.641523 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
                                  00:59:35.280550 ARP, Request who-has y.y.y.51 tell y.y.y.1, length 46
                                  00:59:35.664545 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
                                  00:59:36.688559 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
                                  00:59:49.553602 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
                                  00:59:50.576554 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46
                                  00:59:51.600577 ARP, Request who-has y.y.y.5 tell y.y.y.1, length 46

                                  1 Reply Last reply Reply Quote 0
                                  • S
                                    stephenw10 Netgate Administrator
                                    last edited by Nov 1, 2022, 12:19 AM

                                    You don't have em6 in the bridge also do you? That would break the VLAN.

                                    Bridging to VLANs has always been somewhat shaky!

                                    Can you move that to a separate interface/vswitch? Or do the VLAN tagging in VMWare instead?

                                    Steve

                                    M 1 Reply Last reply Nov 1, 2022, 12:29 AM Reply Quote 0
                                    • S
                                      stephenw10 Netgate Administrator
                                      last edited by Nov 1, 2022, 12:20 AM

                                      You could probably confirm that by running a pcap on the VM.

                                      M 1 Reply Last reply Nov 1, 2022, 12:32 AM Reply Quote 0
                                      • M
                                        mauro.tridici @stephenw10
                                        last edited by Nov 1, 2022, 12:29 AM

                                        @stephenw10

                                        You don't have em6 in the bridge also do you? That would break the VLAN.
                                        Bridging to VLANs has always been somewhat shaky!

                                        Yes, I confirm that only em6.90 is in the bridge :(
                                        I'm sorry...

                                        Can you move that to a separate interface/vswitch? Or do the VLAN tagging in VMWare instead?

                                        I have to check if there is a free interface on the hypervisor, but, at this moment, I'm at home, not at the data center.
                                        I have to think about it. I will let you know.

                                        1 Reply Last reply Reply Quote 0
                                        • M
                                          mauro.tridici @stephenw10
                                          last edited by Nov 1, 2022, 12:32 AM

                                          @stephenw10 running pcap on the VM I can see only the Rapid STP lines and the LLDP with the switch hostname. I think I can confirm that VLAN is broken as you said. (the VM interface is in promiscuous mode)

                                          1 Reply Last reply Reply Quote 0
                                          59 out of 92
                                          • First post
                                            59/92
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                            This community forum collects and processes your personal information.
                                            consent.not_received