Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Can't route public/29 IP block to VMs on lan

    Scheduled Pinned Locked Moved
    Routing and Multi WAN
    4
    21
    376
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • V
      viragomann @MrHedgehog
      last edited by

      @MrHedgehog said in Can't route public/29 IP block to VMs on lan:

      vtnet0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
      description: DMZ
      options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
      ether bc:24:11:54:4e:82
      inet 81.187.2x6.1y7 netmask 0xfffffff8 broadcast 81.187.2x6.1y3

      When you have a /29, it would be better for understanding to obscure the first three octets, but not the last one.

      pppoe0: flags=10088d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
      description: WAN
      options=0
      inet 217.169.1xx.1yy --> 81.187.8x.1y7 netmask 0xffffffff

      So 81.187.8x.1y7 is assigned to the pppoe0. Than you cannot use the same network on another interface as well!

      We were expecting, that you get a single IP via PPPoE and have an additional /29 subnet, which is routed to it by the ISP.

      M 1 Reply Last reply Reply Quote 0
      • M
        MrHedgehog @viragomann
        last edited by

        @viragomann - thanks for the reply and the advice on obscurification (now largely redundant), but I'm not sure I understand:

        xxx.yyy.17.11 --> aaa.bbb.81.187 so the WAN address is xxx.yyy.17.11 and the default route provided by the ISP is aaa.bbb.81.187 on the PPPoE interface.

        My routed IP block is aaa.bbb.206.176/29 and that is assigned to the vtnet0 interface (specifically aaa.bbb.206.177)

        We were expecting, that you get a single IP via PPPoE and have an additional /29 subnet, which is routed to it by the ISP.

        Yes, is that not what I do have? xxx.yyy.17.11 provided by PPPoE and then the /29 is routed to that.

        V 1 Reply Last reply Reply Quote 0
        • V
          viragomann @MrHedgehog
          last edited by

          @MrHedgehog said in Can't route public/29 IP block to VMs on lan:

          xxx.yyy.17.11 --> aaa.bbb.81.187 so the WAN address is xxx.yyy.17.11 and the default route provided by the ISP is aaa.bbb.81.187 on the PPPoE interface.

          Yes, that's correct.

          My routed IP block is aaa.bbb.206.176/29 and that is assigned to the vtnet0 interface (specifically aaa.bbb.206.177)

          Well.

          We were expecting, that you get a single IP via PPPoE and have an additional /29 subnet, which is routed to it by the ISP.

          Yes, is that not what I do have? xxx.yyy.17.11 provided by PPPoE and then the /29 is routed to that.

          So it's fine. It wasn't clear from the above obscuring.

          Now, as mentioned, first of all you should get the communication within the OPT subnet to work.
          Presumed the firewall rules allow it, you should be able to ping the pfSense interface.

          M 1 Reply Last reply Reply Quote 0
          • M
            MrHedgehog @viragomann
            last edited by

            @viragomann - thanks. So from the pfsense box I can ping the DMZ interface (aaa.bbb.206.177)

            [2.7.2-RELEASE][admin@pfsense.lan]/root: ping aaa.bbb.206.177
            PING aaa.bbb.206.177 (aaa.bbb.206.177): 56 data bytes
            64 bytes from aaa.bbb.206.177: icmp_seq=0 ttl=64 time=0.063 ms
            64 bytes from aaa.bbb.206.177: icmp_seq=1 ttl=64 time=0.202 ms
            ^C
            --- aaa.bbb.206.177 ping statistics ---
            2 packets transmitted, 2 packets received, 0.0% packet loss
            round-trip min/avg/max/stddev = 0.063/0.133/0.202/0.069 ms

            But not an IP address in that range assigned to the RPi machine

            [2.7.2-RELEASE][admin@pfsense.lan]/root: ping aaa.bbb.206.178
            PING aaa.bbb.206.178 (aaa.bbb.206.178): 56 data bytes
            ^C
            --- aaa.bbb.206.178 ping statistics ---
            3 packets transmitted, 0 packets received, 100.0% packet loss

            I've put 'allow any any rules' in at the top of the firewall for all interfaces WAN/LAN/DMZ - just as a test, but still no connectivity.

            From the RPi machine, it cannot ping aaa.bbb.206.177.

            Just makes no sense to me. The pfsense instance is running on proxmox and I'm beginning to think there is some issue at the proxmox/virtualisation layer that is blocking the connectivity. I've stopped the firewall that runs within proxmox - but still no joy.

            V 1 Reply Last reply Reply Quote 0
            • V
              viragomann @MrHedgehog
              last edited by

              @MrHedgehog said in Can't route public/29 IP block to VMs on lan:

              But not an IP address in that range assigned to the RPi machine

              [2.7.2-RELEASE][admin@pfsense.lan]/root: ping aaa.bbb.206.178

              Before the pfSense is sending a packet to that IP, it does an ARP lookup. If it gets a proper ARP response it add the IP to its Diagnostic > ARP table.

              Check if you see the respective entry there.

              If not run packet capture on the interface, the destination machine is connected to (DMZ?) and enter ARP at protocol.
              Then start a ping from pfSense. You should see the ARP packets then, at least the ARP request from pfSense.

              If IP is already in the ARP table sniff the ICMP traffic, while you ping the machine.

              M 1 Reply Last reply Reply Quote 0
              • M
                MrHedgehog @viragomann
                last edited by

                @viragomann - thanks again. The arp table does not look as I would expect it to. I've reproduced it below

                [2.7.2-RELEASE][admin@pfsense.lan]/root: arp -a -n | grep '81\.'
? (aaa.bbb.206.177) at bc:24:11:54:4e:82 on vtnet0 permanent [ethernet]

                Now ping the IP and the ARP entry is atted to the table:

                ? (aaa.bbb.206.177) at bc:24:11:54:4e:82 on vtnet0 permanent [ethernet]
? (aaa.bbb.206.178) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 1196 seconds [ethernet]

                However, 2a:8a:1c:ec:d5:20 is not the correct MAC address for the interface on the RPi. I can't find what device corresponds to 2a:8a:1c:ec:d5:20.

                This is the output of tcpdump picking out the arp traffic:

                16:52:49.386790 ARP, Request who-has aaa.bbb.206.178 tell aaa.bbb.206.177, length 28
16:52:49.394704 ARP, Reply aaa.bbb.206.178 is-at 2a:8a:1c:ec:d5:20, length 46

                (An example of proxyarp?)

                What is also strange is that if I manually replace the MAC address for aaa.bbb.206.178 with the correct one, I still cannot ping the device. When looking at tcpdump, I see the ICMP echoo request but no reply.

                The MAC address in the arp table from the RPi showed as 'incomplete' against the aaa.bbb.206.177 address. Again, if I try and delete that and set the MAC entry on the RPi to the correct MAC of the aaa.bbb.206.177 address, that is also still not ping'able.

                Again, thanks for your time.

                johnpozJ V 2 Replies Last reply Reply Quote 0
                • johnpozJ
                  johnpoz LAYER 8 Global Moderator @MrHedgehog
                  last edited by johnpoz

                  @MrHedgehog 2a:8a:1c is unknown.. normally a pi would have a mac that starts with b8:27:eb, atleast the 2 I have currently running do.

                  pi@pi-ntp:~ $ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.3.32  netmask 255.255.255.0  broadcast 192.168.3.255
        inet6 fe80::8849:df44:1c76:170a  prefixlen 64  scopeid 0x20<link>
        ether b8:27:eb:31:70:ab  txqueuelen 1000  (Ethernet)

                  pi@pihole:~ $ ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.3.10  netmask 255.255.255.0  broadcast 192.168.3.255
        inet6 fe80::3829:d17c:fb3d:1b4d  prefixlen 64  scopeid 0x20<link>
        ether b8:27:eb:38:d8:4d  txqueuelen 1000  (Ethernet)

                  An intelligent man is sometimes forced to be drunk to spend time with his fools
                  If you get confused: Listen to the Music Play
                  Please don't Chat/PM me for help, unless mod related
                  SG-4860 24.03 | Lab VMs 2.7.2, 24.03

                  1 Reply Last reply Reply Quote 0
                  • V
                    viragomann @MrHedgehog
                    last edited by

                    @MrHedgehog said in Can't route public/29 IP block to VMs on lan:

                    However, 2a:8a:1c:ec:d5:20 is not the correct MAC address for the interface on the RPi.

                    Possibly a dynamic MAC of Proxmox?

                    johnpozJ M 2 Replies Last reply Reply Quote 0
                    • johnpozJ
                      johnpoz LAYER 8 Global Moderator @viragomann
                      last edited by johnpoz

                      @viragomann how would poxmox have anything to do with a physical pi, if its just a vm - why would he call them pis

                      Maybe a diagram how this is all connected would help..

                      Pfsense is virtual right.. and it tied to a physical interface on the host.. which some other device, a raspberry pi that is what I read when he said RPi - but maybe its some other vm?

                      Been a while since I played with proxmox - but why would it use random macs.. Yeah a drawing would be helpful.. If these are just other vms on the lan side network of pfsense on a virtual switch.. Look at the vm what does it show for its mac? That is what you should be seeing when you arp for it..

                      For example in virtual machine manager.. I can see and change the mac of the vm..

                      mac.jpg

                      Are these VMs on a natted network in proxmox? is the pfsense interface on this vm network, attached to the same virtual switch on the vm host?

                      An intelligent man is sometimes forced to be drunk to spend time with his fools
                      If you get confused: Listen to the Music Play
                      Please don't Chat/PM me for help, unless mod related
                      SG-4860 24.03 | Lab VMs 2.7.2, 24.03

                      S 1 Reply Last reply Reply Quote 0
                      • S
                        SteveITS Rebel Alliance @johnpoz
                        last edited by

                        @johnpoz said in Can't route public/29 IP block to VMs on lan:

                        @viragomann how would poxmox have anything to do with a physical pi,

                        Something else that isn’t the Pi has the IP, if ARP finds the wrong MAC. It’s a question of what.

                        @MrHedgehog Have you tried a different IP in your subnet?

                        Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                        When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                        Upvote 👍 helpful posts!

                        M 1 Reply Last reply Reply Quote 0
                        • M
                          MrHedgehog @viragomann
                          last edited by

                          @viragomann - I am thinking that might be the case and am going to swap out the NVMe card that has the proxmox/pfsense install with another one and do a bare metal install of pfsense on that to strip out the virtualisation layer (this makes it easy to 'rollback' - internet connection being an essential service for my wife!)

                          johnpozJ 1 Reply Last reply Reply Quote 0
                          • M
                            MrHedgehog @SteveITS
                            last edited by

                            @SteveITS - I have tried the other IP addresses in the /29 range. But the odd thing is - all the IPs in that range ARP to the same MAC address, eg from arp -ap on the pfsense machine:

                            ? (aaa.bbb.206.180) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 115 seconds [ethernet]
? (aaa.bbb.206.182) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 130 seconds [ethernet]
? (aaa.bbb.206.183) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 124 seconds [ethernet]
? (aaa.bbb.206.176) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 127 seconds [ethernet]
? (aaa.bbb.206.177) at bc:24:11:54:4e:82 on vtnet0 permanent [ethernet] <- DMZ interface on pfsense
? (aaa.bbb.206.178) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 611 seconds [ethernet]
? (aaa.bbb.206.179) at 2a:8a:1c:ec:d5:20 on vtnet0 expires in 139 seconds [ethernet]

                            I have not idea (at the moment) what 2a:8a:1c:ec:d5:20 corresponds to.

                            1 Reply Last reply Reply Quote 0
                            • johnpozJ
                              johnpoz LAYER 8 Global Moderator @MrHedgehog
                              last edited by

                              @MrHedgehog said in Can't route public/29 IP block to VMs on lan:

                              strip out the virtualisation layer

                              great idea!

                              An intelligent man is sometimes forced to be drunk to spend time with his fools
                              If you get confused: Listen to the Music Play
                              Please don't Chat/PM me for help, unless mod related
                              SG-4860 24.03 | Lab VMs 2.7.2, 24.03

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post