Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Assigning IPs from subnet over GRE to Proxmox VM's

    Scheduled Pinned Locked Moved General pfSense Questions
    19 Posts 2 Posters 715 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      pinguinuk
      last edited by

      Hey guys, I am trying to configure a GRE tunnel on pfSense and route the IPs from GRE to a vLAN connected to Proxmox, does anyone have any ideas on this?

      I have the GRE tunnel active and can see the packets coming in to my gre0 interface, then I have created a vLAN interface and added a IP from the range being sent down the tunnel to it, and then added a IP to a VM. I can ping between pfSense and VM but it seems its acting as a LAN and not sending anything out via GRE as I can not access external networks.

      diagram:
      image.png

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        You can't route that traffic as it's all in the same subnet.

        You could use it dircetly on VLAN20 if the /29 was routed to you over some other smaller subnet.

        You could try bridging the interfaces in pfSense but GRE is a different interfaces type and you may have issues with that.

        You could add the IPs from the /29 as VIPs on pfSense and forward them to the VMs on a different subnet.

        Steve

        P 1 Reply Last reply Reply Quote 0
        • P
          pinguinuk @stephenw10
          last edited by

          @stephenw10 The GRE tunnel is using 172.16.0.0/31 and the /29 is routed over that, once it gets to the pfSense box I have no idea what to do with it to assign each VM it's own /32 from the /29, I want to avoid using NAT entirely.

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            Oh, my mistake. That's fine then. You can just use the /29 subnet on the VLAN20 interface directly. And then set the VMs to use IPs within it with Proxmox set to bridge those to VLAN20.

            P 1 Reply Last reply Reply Quote 0
            • P
              pinguinuk @stephenw10
              last edited by

              @stephenw10 I’ve tried doing that but every time I’ve done it trying every different possible solution I can think of all it will do is refuse to make external connections, for example a ping to 1.1.1.1 with -I to specify the VLAN20 interface will not respond, I can see ICMP requests to 1.1.1.1 in tcpdump for gre0 but it won’t get a reply

              It’s like there’s no external access at all when using the IPs, I can’t ping the IP on the VM from externally either, but if I add it as a virtual IP to an interface it will respond to pings externally, it just doesn’t work when it’s assigned to a VM at all.

              Ping will work from pfSense to the public IP on the VM and I can ping a public IP from the block assigned to pfSense from external and from the VM, it’s just when it’s an IP that’s assigned to a VM that I cannot seem to ping out from the VM or from externally to the VM IP

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                That sounds like a missing route upstream. They are not actually routing the /29 to you.

                If a VIP in the /29 on the GRE does respond it implies the provider have added it directly rather than routed it. You may be able to a ProxyARP VIP for the full /29 to allow it.

                P 1 Reply Last reply Reply Quote 0
                • P
                  pinguinuk @stephenw10
                  last edited by

                  @stephenw10 I'm fairly certain the /29 is routed to me, essentially the reason I have the CloudFlare Magic Transit is because my friend works for CloudFlare and gets to use it, so he can check the backend to make sure it's routed which I'll ask him to check for me now.

                  But this should be routed, the IP's work when added as a VIP (IP Alias mode) to a LAN interface and will load the pfSense login page when accessed from externally in a browser.

                  I'm not entirely sure how to set it up on the VLAN in order to add them to the VM's at this point I've added the start IP of the /29 to a static assignment on VLAN20 and created an interface tagged to VLAN20 on a VM on Proxmox, then under the VM the interface was active named ens19 which I assigned the IP to:

                  ip addr add dev ens19 x.x.x.17/32
                  

                  Then testing connection using the following commands failed:

                  curl --interface ens19 ipinfo.io
                  ping -I ens19 1.1.1.1
                  

                  I have been able to use NAT to port forward HTTP to the VM from one of the IP's and observed that it is accessible from externally, outbound NAT doesn't work however, it's almost as if outbound isn't possible at all.

                  My friend however does have GRE tunnels setup directly to VM's and can do inbound and outbound using the IP's from the same /24 I have the /29 from over CloudFlare Magic Transit so it does work for him, albeit directly on a VM where the GRE ends anyway.

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by stephenw10

                    Ah, OK. Yes, if a VIP added to LAN does respond to external pings then it must be routed to you correctly.

                    You should just be able to add the subnet to the VLAN20 interface in pfSense directly. So probably as the first IP in the subnet: 81.1.212.17/29

                    Then on the VMs add them statically as the other IPs, so 81.1.212.18/29 etc. Set 81.1.212.17 as the default route/gateway for those VMs.

                    P 1 Reply Last reply Reply Quote 0
                    • P
                      pinguinuk @stephenw10
                      last edited by pinguinuk

                      @stephenw10 So i’m still unable to get this to work…

                      I’ve come across this which mentions having a dedicated interface with the subnet attached (I’m using a VLAN for this) and then directly assigning the IPs to the VMs but it mentions bridging the interface to WAN, now in this scenario I believe the GRE interface would be acting as a WAN for the subnet? only issue here is you can’t bridge a GRE to a VLAN…? what can I do here?

                      Additional IP Addresses

                      So far I have:

                      GRE1: local 172.16.0.1 remote 172.16.0.0

                      VLAN20 (igb3.20):
                      Static IPv4: x.x.x.16/29

                      VM (test-vm:m; debian 12):
                      ens3 - 10.128.0.2/16 gw 10.128.0.1
                      ens4 (vlan20) - x.x.x.17/32 gw x.x.x.16

                      outbound NAT disabled on VLAN20 and pfsense in Hybrid Outbound NAT mode

                      I’ve also tested outbound NAT with a virtual ip (proxy arp) on VLAN20 and it can do inbound just fine but the second it comes to translating it outbound it fails with nothing but the outbound request printed in tcpdump for igb3.20 the gre0 interface sees nothing…

                      P 1 Reply Last reply Reply Quote 0
                      • P
                        pinguinuk @pinguinuk
                        last edited by pinguinuk

                        I've just logged on to my laptop now to get some screenshots and details of the configuration to show better detail of how everything is setup.

                        Firstly the GRE Tunnel is configured as such:
                        GRE_settings.png

                        Then the GRE0 interface is configured like this:
                        GRE_IF_settings.png

                        I have the firewall rules for the GRE0 interface set to allow ICMP only at the moment (for Cloudflare health checks on the tunnel):
                        GRE0_FW_rules.png

                        Then VLAN25 is configured as such:
                        VLAN25_vlan_config.png

                        The VLAN25 Interface is configured with x.x.x.17 at the moment but the subnet starts at x.x.x.16 I have tried both of these though...
                        VLAN25_IF_settings.png

                        The Firewall rules for VLAN25 are configured as such:
                        VLAN25_FW_rules.png

                        Now on the test vm the network is configured as such:

                        ip addr add dev enp6s19 x.x.x.18/32
                        ip link set dev enp6s19 up
                        

                        and this is the ip a & ip route output for that machine:

                        root@test-vm:~ # ip a
                        1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
                            link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
                            inet 127.0.0.1/8 scope host lo
                               valid_lft forever preferred_lft forever
                            inet6 ::1/128 scope host noprefixroute
                               valid_lft forever preferred_lft forever
                        2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
                            link/ether bc:24:11:20:1a:aa brd ff:ff:ff:ff:ff:ff
                            altname enp6s18
                            inet 10.128.0.2/16 brd 10.128.255.255 scope global eth0
                               valid_lft forever preferred_lft forever
                            inet6 fe80::be24:11ff:fe20:1aaa/64 scope link
                               valid_lft forever preferred_lft forever
                        4: enp6s19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
                            link/ether bc:24:11:73:54:b6 brd ff:ff:ff:ff:ff:ff
                            inet x.x.x.18/32 scope global enp6s19
                               valid_lft forever preferred_lft forever
                            inet6 fe80::be24:11ff:fe73:54b6/64 scope link
                               valid_lft forever preferred_lft forever
                        root@test-vm:~ # ip route
                        default via 10.128.0.1 dev eth0 proto static
                        10.128.0.0/16 dev eth0 proto kernel scope link src 10.128.0.2
                        root@test-vm:~ #
                        

                        Now initial testing on the test vm to see if that interface can reach the outside world fails:

                        root@test-vm:~ # curl --interface enp6s19 ipinfo.io
                        curl: (7) Failed to connect to ipinfo.io port 80 after 3081 ms: Couldn't connect to server
                        root@test-vm:~ # ping -I enp6s19 1.1.1.1
                        PING 1.1.1.1 (1.1.1.1) from 81.31.212.18 enp6s19: 56(84) bytes of data.
                        ^C
                        --- 1.1.1.1 ping statistics ---
                        7 packets transmitted, 0 received, 100% packet loss, time 6145ms
                        pipe 4
                        root@test-vm:~ #
                        

                        Leaving ping -I enp6s19 1.1.1.1 running on the test vm and watching tcpdump -i gre0 and tcpdump -i igb3.25 output on the pfsense box shows the following...

                        For igb3.25:

                        root@pfSense:~ # tcpdump -i igb3.25
                        tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
                        listening on igb3.25, link-type EN10MB (Ethernet), snapshot length 262144 bytes
                        11:43:30.627432 ARP, Request who-has one.one.one.one tell x.x.x.18, length 42
                        11:43:31.651369 ARP, Request who-has one.one.one.one tell x.x.x.18, length 42
                        11:43:32.675386 ARP, Request who-has one.one.one.one tell x.x.x.18, length 42
                        11:43:33.699424 ARP, Request who-has one.one.one.one tell x.x.x.18, length 42
                        11:43:34.723326 ARP, Request who-has one.one.one.one tell x.x.x.18, length 42
                        11:43:35.747338 ARP, Request who-has one.one.one.one tell x.x.x.18, length 42
                        11:43:36.772000 ARP, Request who-has one.one.one.one tell x.x.x.18, length 42
                        11:43:37.795302 ARP, Request who-has one.one.one.one tell x.x.x.18, length 42
                        ^C
                        8 packets captured
                        8 packets received by filter
                        0 packets dropped by kernel
                        root@pfSense:~ #
                        

                        You can see that this is doing "something" but it only appears to be sending ARP requests and nothing is responding to those requests...

                        Now running tcpdump -i gre0 not icmp | grep "x.x.x.18" on pfSense gets a little bit interesting, you can't see any of the ICMP requests heading outbound on the GRE0 interface at all, BUT! you can see the inbound requests from my mail server which I performed a traceroute on to "x.x.x.18" (shown further down):

                        root@pfSense:~ # tcpdump -i gre0 not icmp | grep "x.x.x.18"
                        tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
                        listening on gre0, link-type NULL (BSD loopback), snapshot length 262144 bytes
                        11:46:29.871382 IP scan-52g.shadowserver.org.35479 > x.x.x.18.9001: Flags [S], seq 599764770, win 65535, length 0
                        11:46:31.516766 IP 222.89.13.82.50919 > x.x.x.18.telnet: Flags [S], seq 1361040402, win 58055, options [mss 536], length 0
                        11:46:32.701287 IP mail.redacted.56399 > x.x.x.18.33471: UDP, length 32
                        11:46:32.701307 IP mail.redacted.51094 > x.x.x.18.33472: UDP, length 32
                        11:46:32.701310 IP mail.redacted.32176 > x.x.x.18.33468: UDP, length 32
                        11:46:32.702410 IP mail.redacted.23634 > x.x.x.18.33470: UDP, length 32
                        11:46:33.992008 IP mail.redacted.3235 > x.x.x.18.33477: UDP, length 32
                        11:46:33.992030 IP mail.redacted.20704 > x.x.x.18.33473: UDP, length 32
                        11:46:33.992134 IP mail.redacted.65246 > x.x.x.18.33474: UDP, length 32
                        11:46:33.992148 IP mail.redacted.25379 > x.x.x.18.33479: UDP, length 32
                        

                        And here you can see the failed ping test and the traceroute from my mail server (external) to x.x.x.18:

                        toor@mail:~ % ping x.x.x.18
                        PING x.x.x.18 (x.x.x.18) 56(84) bytes of data.
                        ^C
                        --- x.x.x.18 ping statistics ---
                        7 packets transmitted, 0 received, 100% packet loss, time 6121ms
                        
                        toor@mail:~ % traceroute x.x.x.18
                        traceroute to x.x.x.18 (x.x.x.18), 30 hops max, 60 byte packets
                         1  10.207.6.111 (10.207.6.111)  0.208 ms  0.086 ms  0.066 ms
                         2  10.207.35.20 (10.207.35.20)  0.249 ms 10.207.35.19 (10.207.35.19)  0.240 ms 10.207.35.20 (10.207.35.20)  0.149 ms
                         3  10.207.32.2 (10.207.32.2)  0.157 ms  0.238 ms 10.207.32.1 (10.207.32.1)  0.203 ms
                         4  lo0-0.gw2.lon1.gb.linode.com (109.74.207.102)  0.455 ms  0.311 ms lo0-0.gw1.lon1.gb.linode.com (109.74.207.101)  0.548 ms
                         5  ae17.r02.lon03.ien.netarch.akamai.com (23.210.50.24)  1.294 ms ae20.r21.lon01.ien.netarch.akamai.com (23.210.48.106)  0.703 ms ae19.r22.lon01.ien.netarch.akamai.com (23.210.48.108)  0.649 ms
                         6  ae5.r22.lon01.mag.netarch.akamai.com (23.197.64.70)  1.072 ms ae4.r22.lon01.mag.netarch.akamai.com (23.197.64.66)  0.934 ms  0.772 ms
                         7  ae2.r23.lon01.ien.netarch.akamai.com (23.197.64.75)  0.665 ms  0.637 ms 141.101.71.61 (141.101.71.61)  2.760 ms
                         8  a23-210-48-145.deploy.static.akamaitechnologies.com (23.210.48.145)  24.593 ms 172.70.88.254 (172.70.88.254)  1.538 ms a23-210-48-235.deploy.static.akamaitechnologies.com (23.210.48.235)  1.309 ms
                         9  141.101.71.105 (141.101.71.105)  5.685 ms 141.101.71.47 (141.101.71.47)  1.666 ms 141.101.71.91 (141.101.71.91)  1.220 ms
                        10  141.101.70.91 (141.101.70.91)  0.873 ms 172.70.161.124 (172.70.161.124)  1.168 ms 141.101.70.45 (141.101.70.45)  1.573 ms
                        11  172.69.193.224 (172.69.193.224)  1.474 ms 172.70.84.53 (172.70.84.53)  1.268 ms 172.70.89.11 (172.70.89.11)  1.378 ms
                        12  172.69.43.38 (172.69.43.38)  1.472 ms * 172.69.193.134 (172.69.193.134)  1.361 ms
                        13  * * *
                        14  * * *
                        15  * * *
                        16  * * *
                        17  * * *
                        18  * * *
                        19  * * *
                        20  * * *
                        21  * * *
                        22  * * *
                        23  * * *
                        24  * * *
                        25  * * *
                        26  * * *
                        27  * * *
                        28  * * *
                        29  * * *
                        30  * * *
                        toor@mail:~ %
                        

                        I am completely out of ideas at this point, as it's entirely unclear why this isn't just working at this point?

                        I do hope this helps expand on how my setup is running though.

                        EDIT: performed a further test and x.x.x.17 is pingable from my mail server

                        toor@mail:~ % ping x.x.x.17
                        PING x.x.x.17 (x.x.x.17) 56(84) bytes of data.
                        64 bytes from x.x.x.17: icmp_seq=1 ttl=53 time=18.0 ms
                        64 bytes from x.x.x.17: icmp_seq=2 ttl=53 time=18.8 ms
                        ^C
                        --- x.x.x.17 ping statistics ---
                        2 packets transmitted, 2 received, 0% packet loss, time 1001ms
                        rtt min/avg/max/mdev = 17.973/18.385/18.797/0.412 ms
                        toor@mail:~ %
                        

                        But not the x.x.x.18 ip assigned to the vm...

                        EDIT 2: Decided it may be helpful to show the routing table on pfSense here as well

                        y.y.y.y is my ISP related addresses
                        x.x.x.x is my GRE tunnel/Magic Transit related addresses

                        root@pfSense:~ # netstat -rn
                        Routing tables
                        
                        Internet:
                        Destination        Gateway            Flags     Netif Expire
                        default            y.y.y.1        UGS        igb0
                        10.0.0.0/16        link#2             U          igb1
                        10.0.0.1           link#7             UHS         lo0
                        10.2.0.0/16        link#3             U          igb2
                        10.2.0.1           link#7             UHS         lo0
                        10.4.0.0/16        link#4             U          igb3
                        10.4.0.1           link#7             UHS         lo0
                        10.7.0.0/24        link#10            U       tun_wg0
                        10.7.0.2           link#7             UHS         lo0
                        10.32.0.0/16       link#13            U       igb3.10
                        10.32.0.1          link#7             UHS         lo0
                        10.64.0.0/16       link#14            U       igb3.15
                        10.64.0.1          link#7             UHS         lo0
                        10.69.0.0/24       link#11            U       tun_wg1
                        10.69.0.1          link#7             UHS         lo0
                        10.70.0.0/24       link#12            U       tun_wg2
                        10.70.0.1          link#7             UHS         lo0
                        10.128.0.0/24      link#15            U       igb3.20
                        10.128.0.1         link#7             UHS         lo0
                        x.x.x.16/29    link#17            U       igb3.25
                        x.x.x.17       link#7             UHS         lo0
                        x.x.x.0/23     link#1             U          igb0
                        y.y.y.1        link#1             UHS        igb0
                        y.y.y.152      link#7             UHS         lo0
                        127.0.0.1          link#7             UH          lo0
                        172.16.0.0         link#16            UH         gre0
                        172.16.0.0/31      172.16.0.0         UGS         lo0
                        172.16.0.0/31      172.16.0.1         UGS         lo0
                        172.16.0.1         link#7             UHS         lo0
                        172.17.0.0/30      172.17.0.2         UGS         lo0
                        172.19.0.0/31      172.19.0.1         UGS         lo0
                        172.19.0.0/24      172.19.0.2         UGS         lo0
                        194.168.4.100      link#1             UHS        igb0
                        194.168.8.100      link#1             UHS        igb0
                        
                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          If the /29 is routed to you then you should just be able to use it on VLAN20 as you have done. And it looks like it must be routed to you because a VIP on LAN could not respond to ARP requests on WAN/GRE.

                          Can you ping the .16 IP address you assigned to the VLAN20 interface from an external source? Assuming you have rules to pass that on GRE.

                          If that works then VMs using IPs in that subnet should also work.

                          P 1 Reply Last reply Reply Quote 0
                          • P
                            pinguinuk @stephenw10
                            last edited by

                            @stephenw10 Sorry I didn't see your reply, I have edited my previous reply with output of a ping from my mail server (external) to the .17 IP on VLAN25 (I am using a new vlan; 25 is the one for this test now)

                            So the .17 IP address does indeed ping from externally just fine, it's assigned directly to the interface (no virtual IP is used here):

                            toor@mail:~ % ping x.x.x.17
                            PING x.x.x.17 (x.x.x.17) 56(84) bytes of data.
                            64 bytes from x.x.x.17: icmp_seq=1 ttl=53 time=18.0 ms
                            64 bytes from x.x.x.17: icmp_seq=2 ttl=53 time=18.8 ms
                            ^C
                            --- x.x.x.17 ping statistics ---
                            2 packets transmitted, 2 received, 0% packet loss, time 1001ms
                            rtt min/avg/max/mdev = 17.973/18.385/18.797/0.412 ms
                            toor@mail:~ %
                            

                            I also provided the routing table from pfSense as an edit to my previous response just now as well :)

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              Ok you need to set the subnet mask to /29 on the VMs too. And set a gateway/default route on the VMs as the pfSense interface IP so they are able to route out.

                              P 1 Reply Last reply Reply Quote 0
                              • P
                                pinguinuk @stephenw10
                                last edited by

                                @stephenw10 Okay so I've deleted the IP and re-added it as a /29 on the VM:

                                ip addr del dev enp6s19 x.x.x.18/32
                                ip addr add dev enp6s19 x.x.x.18/29
                                ip route add default via x.x.x.17
                                

                                I set the default route via x.x.x.17 (vlan25 gw ip), here is the output from route -n:

                                root@test-vm:~ # route -n
                                Kernel IP routing table
                                Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
                                0.0.0.0         x.x.x.17    0.0.0.0         UG    0      0        0 enp6s19
                                10.128.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth0
                                x.x.x.16    0.0.0.0         255.255.255.248 U     0      0        0 enp6s19
                                root@test-vm:~ #
                                

                                Now it feels like we are getting somewhat closer here...?

                                I lost connectivity to the VM over SSH to the VLAN20 IP 10.128.0.2 but I am now able to SSH to x.x.x.18 which is assigned to the VM and ping it from my MacBook on LAN1:

                                rtbt@Toms-MacBook-Pro ~ % ping x.x.x.18
                                PING x.x.x.18 (x.x.x.18): 56 data bytes
                                64 bytes from x.x.x.18: icmp_seq=0 ttl=63 time=8.685 ms
                                64 bytes from x.x.x.18: icmp_seq=1 ttl=63 time=9.524 ms
                                ^C
                                --- x.x.x.18 ping statistics ---
                                2 packets transmitted, 2 packets received, 0.0% packet loss
                                round-trip min/avg/max/stddev = 8.685/9.104/9.524/0.419 ms
                                rtbt@Toms-MacBook-Pro ~ %
                                

                                So internally the IP addresses are working, but there's still no external connectivity whatsoever from the VM besides to the VLAN20 gw or the VLAN25 gw:

                                root@test-vm:~ # ping 1.1.1.1
                                PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
                                ^C
                                --- 1.1.1.1 ping statistics ---
                                2 packets transmitted, 0 received, 100% packet loss, time 1026ms
                                
                                root@test-vm:~ # ping -I enp6s19 1.1.1.1
                                PING 1.1.1.1 (1.1.1.1) from x.x.x.18 enp6s19: 56(84) bytes of data.
                                ^C
                                --- 1.1.1.1 ping statistics ---
                                3 packets transmitted, 0 received, 100% packet loss, time 2047ms
                                
                                root@test-vm:~ # curl ipinfo.io
                                ^C
                                root@test-vm:~ # curl --interface x.x.x.18 ipinfo.io
                                ^C
                                root@test-vm:~ # curl --interface enp6s19 ipinfo.io
                                ^C
                                root@test-vm:~ # ping 10.128.0.1
                                PING 10.128.0.1 (10.128.0.1) 56(84) bytes of data.
                                64 bytes from 10.128.0.1: icmp_seq=1 ttl=64 time=0.274 ms
                                64 bytes from 10.128.0.1: icmp_seq=2 ttl=64 time=0.322 ms
                                ^C
                                --- 10.128.0.1 ping statistics ---
                                2 packets transmitted, 2 received, 0% packet loss, time 1032ms
                                rtt min/avg/max/mdev = 0.274/0.298/0.322/0.024 ms
                                root@test-vm:~ # ping x.x.x.17
                                PING x.x.x.17 (x.x.x.17) 56(84) bytes of data.
                                64 bytes from x.x.x.17: icmp_seq=1 ttl=64 time=0.283 ms
                                ^C
                                --- x.x.x.17 ping statistics ---
                                1 packets transmitted, 1 received, 0% packet loss, time 0ms
                                rtt min/avg/max/mdev = 0.283/0.283/0.283/0.000 ms
                                root@test-vm:~ #
                                
                                P 1 Reply Last reply Reply Quote 0
                                • P
                                  pinguinuk @pinguinuk
                                  last edited by

                                  So some new information has been uncovered here...

                                  I performed a traceroute to 1.1.1.1 and the Cloudflare side of the GRE tunnel responded!

                                  172.16.0.0 (remote; cloudflare end)
                                  172.16.0.1 (local; pfsense end)

                                  root@test-vm:~ # traceroute 1.1.1.1
                                  traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
                                   1  172.16.0.0 (172.16.0.0)  15.463 ms  15.431 ms  15.400 ms
                                   2  * * *
                                   3  * * *
                                   4  * * *
                                   5  * * *
                                   6  * * *
                                   7  * * *
                                   8  * * *
                                   9  * * *
                                  10  * * *
                                  11  * * *
                                  12  * * *
                                  13  * * *
                                  14  * * *
                                  15  * * *
                                  16  * * *
                                  17  * * *
                                  18  * * *
                                  19  * * *
                                  20  * * *
                                  21  * * *
                                  22  * * *
                                  23  * * *
                                  24  * * *
                                  25  * * *
                                  26  * * *
                                  27  * * *
                                  28  * * *
                                  29  * * *
                                  30  * * *
                                  root@test-vm:~ #
                                  

                                  During the test I performed a tcpdump on gre0 and seen the following:

                                  13:08:31.455182 IP 172.16.0.1.19976 > one.one.one.one.33435: UDP, length 32
                                  13:08:31.455215 IP 172.16.0.1.26370 > one.one.one.one.33434: UDP, length 32
                                  13:08:31.455246 IP 172.16.0.1.57490 > one.one.one.one.33436: UDP, length 32
                                  13:08:31.455272 IP 172.16.0.1.19245 > one.one.one.one.33438: UDP, length 32
                                  13:08:31.455288 IP 172.16.0.1.28308 > one.one.one.one.33439: UDP, length 32
                                  13:08:31.455302 IP 172.16.0.1.17065 > one.one.one.one.33440: UDP, length 32
                                  13:08:31.455314 IP 172.16.0.1.25154 > one.one.one.one.33437: UDP, length 32
                                  13:08:31.455328 IP 172.16.0.1.16824 > one.one.one.one.33441: UDP, length 32
                                  13:08:31.455416 IP 172.16.0.1.58503 > one.one.one.one.33442: UDP, length 32
                                  13:08:31.455442 IP 172.16.0.1.41873 > one.one.one.one.33443: UDP, length 32
                                  13:08:31.455456 IP 172.16.0.1.58879 > one.one.one.one.33444: UDP, length 32
                                  13:08:31.455469 IP 172.16.0.1.35433 > one.one.one.one.33445: UDP, length 32
                                  13:08:31.455538 IP 172.16.0.1.12616 > one.one.one.one.33447: UDP, length 32
                                  13:08:31.455572 IP 172.16.0.1.11424 > one.one.one.one.33448: UDP, length 32
                                  13:08:31.455584 IP 172.16.0.1.35466 > one.one.one.one.33446: UDP, length 32
                                  13:08:31.455597 IP 172.16.0.1.9850 > one.one.one.one.33449: UDP, length 32
                                  13:08:31.466959 IP 172.16.0.1.5969 > one.one.one.one.33452: UDP, length 32
                                  13:08:31.466993 IP 172.16.0.1.30486 > one.one.one.one.33450: UDP, length 32
                                  13:08:31.467017 IP 172.16.0.1.35717 > one.one.one.one.33451: UDP, length 32
                                  

                                  So to me this feels like the traffic is now traversing the GRE tunnel and landing at Cloudflare?

                                  Could this potentially now be an issue on the Cloudflare end considering the above?

                                  P 1 Reply Last reply Reply Quote 0
                                  • P
                                    pinguinuk @pinguinuk
                                    last edited by

                                    I think I realised why traffic is being sent down the tunnel:
                                    Screenshot 2024-08-10 at 13.19.36.png

                                    I have the any/any rule forcing traffic down the tunnel which if I remove then traffic flows outbound but only through my ISP IP and doesn't use the assigned IP from the /29

                                    P 1 Reply Last reply Reply Quote 0
                                    • P
                                      pinguinuk @pinguinuk
                                      last edited by pinguinuk

                                      Another quick progress report, I started a web server on the test vm and opened port 80 for x.x.x.18 on the gre0 interface and I am able to access the web server externally:

                                      toor@mail:~ % curl http://x.x.x.18
                                      <p>This is the test VM!!</p>
                                      toor@mail:~ %
                                      

                                      I just cannot get outbound working at all! :(

                                      P 1 Reply Last reply Reply Quote 0
                                      • P
                                        pinguinuk @pinguinuk
                                        last edited by

                                        Okay so I've got this working!

                                        I disabled all Outbound NAT rules for GRE0 and VLAN25 and added two 'NO NAT' outbound rules to the GRE0 and VLAN25 interface and after testing on the VM I can see that it's working now!!!

                                        1 Reply Last reply Reply Quote 1
                                        • stephenw10S
                                          stephenw10 Netgate Administrator
                                          last edited by

                                          Nice! Yeah there's a lot of ways to get that to fail! Sounds like you removed them all though. 😉

                                          1 Reply Last reply Reply Quote 1
                                          • First post
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.