Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Can't route public/29 IP block to VMs on lan

    Scheduled Pinned Locked Moved
    Routing and Multi WAN
    4
    21
    373
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      MrHedgehog @viragomann
      last edited by

      @viragomann - thanks. So from the pfsense box I can ping the DMZ interface (aaa.bbb.206.177)

      [2.7.2-RELEASE][admin@pfsense.lan]/root: ping aaa.bbb.206.177
      PING aaa.bbb.206.177 (aaa.bbb.206.177): 56 data bytes
      64 bytes from aaa.bbb.206.177: icmp_seq=0 ttl=64 time=0.063 ms
      64 bytes from aaa.bbb.206.177: icmp_seq=1 ttl=64 time=0.202 ms
      ^C
      --- aaa.bbb.206.177 ping statistics ---
      2 packets transmitted, 2 packets received, 0.0% packet loss
      round-trip min/avg/max/stddev = 0.063/0.133/0.202/0.069 ms

      But not an IP address in that range assigned to the RPi machine

      [2.7.2-RELEASE][admin@pfsense.lan]/root: ping aaa.bbb.206.178
      PING aaa.bbb.206.178 (aaa.bbb.206.178): 56 data bytes
      ^C
      --- aaa.bbb.206.178 ping statistics ---
      3 packets transmitted, 0 packets received, 100.0% packet loss

      I've put 'allow any any rules' in at the top of the firewall for all interfaces WAN/LAN/DMZ - just as a test, but still no connectivity.

      From the RPi machine, it cannot ping aaa.bbb.206.177.

      Just makes no sense to me. The pfsense instance is running on proxmox and I'm beginning to think there is some issue at the proxmox/virtualisation layer that is blocking the connectivity. I've stopped the firewall that runs within proxmox - but still no joy.

      V 1 Reply Last reply Reply Quote 0
      • V
        viragomann @MrHedgehog
        last edited by

        @MrHedgehog said in Can't route public/29 IP block to VMs on lan:

        But not an IP address in that range assigned to the RPi machine

        [2.7.2-RELEASE][admin@pfsense.lan]/root: ping aaa.bbb.206.178

        Before the pfSense is sending a packet to that IP, it does an ARP lookup. If it gets a proper ARP response it add the IP to its Diagnostic > ARP table.

        Check if you see the respective entry there.

        If not run packet capture on the interface, the destination machine is connected to (DMZ?) and enter ARP at protocol.
        Then start a ping from pfSense. You should see the ARP packets then, at least the ARP request from pfSense.

        If IP is already in the ARP table sniff the ICMP traffic, while you ping the machine.

        M 1 Reply Last reply Reply Quote 0
        • M
          MrHedgehog @viragomann
          last edited by

          @viragomann - thanks again. The arp table does not look as I would expect it to. I've reproduced it below

          [2.7.2-RELEASE][admin@pfsense.lan]/root: arp -a -n | grep '81\.'
? (aaa.bbb.206.177) at bc:24:11:54:4e:82 on vtnet0 permanent [ethernet]

          Now ping the IP and the ARP entry is atted to the table:

          ? (aaa.bbb.206.177) at bc:24:11:54:4e:82 on vtnet0 permanent [ethernet]
? (aaa.bbb.206.178) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 1196 seconds [ethernet]

          However, 2a:8a:1c:ec:d5:20 is not the correct MAC address for the interface on the RPi. I can't find what device corresponds to 2a:8a:1c:ec:d5:20.

          This is the output of tcpdump picking out the arp traffic:

          16:52:49.386790 ARP, Request who-has aaa.bbb.206.178 tell aaa.bbb.206.177, length 28
16:52:49.394704 ARP, Reply aaa.bbb.206.178 is-at 2a:8a:1c:ec:d5:20, length 46

          (An example of proxyarp?)

          What is also strange is that if I manually replace the MAC address for aaa.bbb.206.178 with the correct one, I still cannot ping the device. When looking at tcpdump, I see the ICMP echoo request but no reply.

          The MAC address in the arp table from the RPi showed as 'incomplete' against the aaa.bbb.206.177 address. Again, if I try and delete that and set the MAC entry on the RPi to the correct MAC of the aaa.bbb.206.177 address, that is also still not ping'able.

          Again, thanks for your time.

          johnpozJ V 2 Replies Last reply Reply Quote 0
          • johnpozJ
            johnpoz LAYER 8 Global Moderator @MrHedgehog
            last edited by johnpoz

            @MrHedgehog 2a:8a:1c is unknown.. normally a pi would have a mac that starts with b8:27:eb, atleast the 2 I have currently running do.

            pi@pi-ntp:~ $ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.3.32  netmask 255.255.255.0  broadcast 192.168.3.255
        inet6 fe80::8849:df44:1c76:170a  prefixlen 64  scopeid 0x20<link>
        ether b8:27:eb:31:70:ab  txqueuelen 1000  (Ethernet)

            pi@pihole:~ $ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.3.10  netmask 255.255.255.0  broadcast 192.168.3.255
        inet6 fe80::3829:d17c:fb3d:1b4d  prefixlen 64  scopeid 0x20<link>
        ether b8:27:eb:38:d8:4d  txqueuelen 1000  (Ethernet)

            An intelligent man is sometimes forced to be drunk to spend time with his fools
            If you get confused: Listen to the Music Play
            Please don't Chat/PM me for help, unless mod related
            SG-4860 24.03 | Lab VMs 2.7.2, 24.03

            1 Reply Last reply Reply Quote 0
            • V
              viragomann @MrHedgehog
              last edited by

              @MrHedgehog said in Can't route public/29 IP block to VMs on lan:

              However, 2a:8a:1c:ec:d5:20 is not the correct MAC address for the interface on the RPi.

              Possibly a dynamic MAC of Proxmox?

              johnpozJ M 2 Replies Last reply Reply Quote 0
              • johnpozJ
                johnpoz LAYER 8 Global Moderator @viragomann
                last edited by johnpoz

                @viragomann how would poxmox have anything to do with a physical pi, if its just a vm - why would he call them pis

                Maybe a diagram how this is all connected would help..

                Pfsense is virtual right.. and it tied to a physical interface on the host.. which some other device, a raspberry pi that is what I read when he said RPi - but maybe its some other vm?

                Been a while since I played with proxmox - but why would it use random macs.. Yeah a drawing would be helpful.. If these are just other vms on the lan side network of pfsense on a virtual switch.. Look at the vm what does it show for its mac? That is what you should be seeing when you arp for it..

                For example in virtual machine manager.. I can see and change the mac of the vm..

                mac.jpg

                Are these VMs on a natted network in proxmox? is the pfsense interface on this vm network, attached to the same virtual switch on the vm host?

                An intelligent man is sometimes forced to be drunk to spend time with his fools
                If you get confused: Listen to the Music Play
                Please don't Chat/PM me for help, unless mod related
                SG-4860 24.03 | Lab VMs 2.7.2, 24.03

                S 1 Reply Last reply Reply Quote 0
                • S
                  SteveITS Rebel Alliance @johnpoz
                  last edited by

                  @johnpoz said in Can't route public/29 IP block to VMs on lan:

                  @viragomann how would poxmox have anything to do with a physical pi,

                  Something else that isn’t the Pi has the IP, if ARP finds the wrong MAC. It’s a question of what.

                  @MrHedgehog Have you tried a different IP in your subnet?

                  Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                  When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                  Upvote 👍 helpful posts!

                  M 1 Reply Last reply Reply Quote 0
                  • M
                    MrHedgehog @viragomann
                    last edited by

                    @viragomann - I am thinking that might be the case and am going to swap out the NVMe card that has the proxmox/pfsense install with another one and do a bare metal install of pfsense on that to strip out the virtualisation layer (this makes it easy to 'rollback' - internet connection being an essential service for my wife!)

                    johnpozJ 1 Reply Last reply Reply Quote 0
                    • M
                      MrHedgehog @SteveITS
                      last edited by

                      @SteveITS - I have tried the other IP addresses in the /29 range. But the odd thing is - all the IPs in that range ARP to the same MAC address, eg from arp -ap on the pfsense machine:

                      ? (aaa.bbb.206.180) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 115 seconds [ethernet]
? (aaa.bbb.206.182) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 130 seconds [ethernet]
? (aaa.bbb.206.183) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 124 seconds [ethernet]
? (aaa.bbb.206.176) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 127 seconds [ethernet]
? (aaa.bbb.206.177) at bc:24:11:54:4e:82 on vtnet0 permanent [ethernet] <- DMZ interface on pfsense
? (aaa.bbb.206.178) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 611 seconds [ethernet]
? (aaa.bbb.206.179) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 139 seconds [ethernet]

                      I have not idea (at the moment) what 2a:8a:1c:ec:d5:20 corresponds to.

                      1 Reply Last reply Reply Quote 0
                      • johnpozJ
                        johnpoz LAYER 8 Global Moderator @MrHedgehog
                        last edited by

                        @MrHedgehog said in Can't route public/29 IP block to VMs on lan:

                        strip out the virtualisation layer

                        great idea!

                        An intelligent man is sometimes forced to be drunk to spend time with his fools
                        If you get confused: Listen to the Music Play
                        Please don't Chat/PM me for help, unless mod related
                        SG-4860 24.03 | Lab VMs 2.7.2, 24.03

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post