Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Pfsense 2.4.x routes broken/weird after some time. Working on 2.3.x.

    Scheduled Pinned Locked Moved OpenVPN
    6 Posts 2 Posters 1.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • F
      fogelholk
      last edited by

      I've been reading some threads with what would seem like similar problems, but didn't feel like hijacking their threads if it would result in not being the same issue.

      So here is is. I have a few static routes set up with rules allowing traffic between different VPN connections.

      Whenever, what seems to be consistent, my ISP or the VPN provider I connect through has a random disconnect, my static routes changes from the VPN interface to the localhost interface by checking netstat -rn which results in my roadwarrior clients not being able to utilize the static routes I've set up.

      The scenario that is working in pfSense 2.3, even through random disconnects, is that my roadwarrior client (OpenVPN) can connect through VPN connections I have set up on my pfSense to my work place.
      In 2.4.x (currently 2.4.2) this connection also works perfectly fine until, for example, my ISP has a random disconnect. After this my roadwarrior client can not send traffic through the VPN tunnel I have set up from pfSense to my work place.

      I hope I haven't obfuscated too much for it to be readable, but if a dev wants the real information, I'd be happy to send it privately if needed. If you need more information, just ask away! I really want to solve this and keep using pfSense 2.4, and not downgrade to 2.3 or restart pfSense weekly to have my roadwarrior working.

      Here's netstat when everything is working on pfSense 2.4.2:

      Internet:
      Destination        Gateway            Flags     Netif Expire
      default            zzz.zzz.zzz.205    UGS         lo0
      10.0.11.0/24       10.0.11.1          UGS         lo0
      10.0.11.1          link#8             UHS         lo0
      10.0.11.2          link#8             UH       ovpns1
      tt.tt.ttt.145      xxx.xxx.51.1       UGHS       igb1
      yy.yyy.y.0/21      172.22.233.131     UGS      ovpnc3
      yy.yyy.y.20/31     xxx.xxx.51.1       UGS        igb1
      127.0.0.1          link#3             UH          lo0
      172.21.0.0/16      172.22.233.131     UGS      ovpnc3
      172.22.0.0/16      172.22.233.131     UGS      ovpnc3
      172.22.233.128/25  172.22.233.129     UGS      ovpnc3
      172.22.233.129     link#10            UH       ovpnc3
      172.22.233.131     link#10            UHS         lo0
      192.168.11.0/24    link#1             U          igb0
      192.168.11.1       link#1             UHS         lo0
      xxx.xxx.51.0/25    link#2             U          igb1
      xxx.xxx.51.94      link#2             UHS         lo0
      sss.sss.0.10       xxx.xxx.51.1       UGHS       igb1
      uuu.uuu.uuu.2      xxx.xxx.51.1       UGHS       igb1
      zzz.zzz.zzz.192/26 zzz.zzz.zzz.193    UGS      ovpnc2
      zzz.zzz.zzz.193    link#9             UH       ovpnc2
      zzz.zzz.zzz.205    link#9             UHS         lo0
      vvv.v.vv.0/23      172.22.233.131     UGS      ovpnc3
      vvv.v.vv.231/32    xxx.xxx.51.1       UGS        igb1
      www.ww.ww.90       xxx.xxx.51.1       UGHS       igb1
      

      And here's netstat when it's Not working on pfsense 2.4.2:

      Internet:
      Destination        Gateway            Flags     Netif Expire
      default            zzz.zzz.zzz.144    UGS         lo0
      10.0.11.0/24       10.0.11.1          UGS         lo0
      10.0.11.1          link#9             UHS         lo0
      10.0.11.2          link#9             UH       ovpns1
      tt.tt.ttt.145      xxx.xxx.51.1       UGHS       igb1
      yy.yyy.y.0/21      172.22.233.131     UGS         lo0
      yy.yyy.y.20/31     xxx.xxx.51.1       UGS        igb1
      127.0.0.1          link#3             UH          lo0
      172.21.0.0/16      172.22.233.131     UGS         lo0
      172.22.0.0/16      172.22.233.131     UGS         lo0
      172.22.233.128/25  172.22.233.129     UGS      ovpnc3
      172.22.233.129     link#11            UH       ovpnc3
      172.22.233.131     link#11            UHS         lo0
      192.168.11.0/24    link#1             U          igb0
      192.168.11.1       link#1             UHS         lo0
      xxx.xxx.51.0/25    link#2             U          igb1
      xxx.xxx.51.94      link#2             UHS         lo0
      sss.sss.0.10       xxx.xxx.51.1       UGHS       igb1
      uuu.uuu.uuu.2      xxx.xxx.51.1       UGHS       igb1
      zzz.zzz.zzz.128/26 zzz.zzz.zzz.129    UGS      ovpnc2
      zzz.zzz.zzz.129    link#10            UH       ovpnc2
      zzz.zzz.zzz.144    link#10            UHS         lo0
      vvv.v.vv.0/23      172.22.233.131     UGS         lo0
      vvv.v.vv.231/32    xxx.xxx.51.1       UGS        igb1
      www.ww.ww.90       xxx.xxx.51.1       UGHS       igb1
      

      And here's an ifconfig output, also onfuscated to hell:

      igb0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
              options=6403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6>ether mm:mm:mm:mm:mm:06
              hwaddr mm:mm:mm:mm:mm:06
              inet6 fe80::12c3:7bff:fe47:e006%igb0 prefixlen 64 scopeid 0x1
              inet 192.168.11.1 netmask 0xffffff00 broadcast 192.168.11.255
              nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
              status: active
      igb1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
              options=6403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6>ether mm:mm:mm:mm:mm:07
              hwaddr mm:mm:mm:mm:mm:07
              inet6 fe80::12c3:7bff:fe47:e007%igb1 prefixlen 64 scopeid 0x2
              inet xxx.xxx.51.94 netmask 0xffffff80 broadcast xxx.xxx.51.127
              nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
              status: active
      lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
              options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet6 ::1 prefixlen 128
              inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
              inet 127.0.0.1 netmask 0xff000000
              nd6 options=21 <performnud,auto_linklocal>groups: lo
      enc0: flags=41 <up,running>metric 0 mtu 1536
              nd6 options=21 <performnud,auto_linklocal>groups: enc
      pflog0: flags=100 <promisc>metric 0 mtu 33160
              groups: pflog
      pfsync0: flags=0<> metric 0 mtu 1500
              groups: pfsync
              syncpeer: 224.0.0.240 maxupd: 128 defer: on
              syncok: 1
      ovpns1: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
              options=80000 <linkstate>inet6 fe80::12c3:7bff:fe47:e006%ovpns1 prefixlen 64 scopeid 0x8
              inet 10.0.11.1 --> 10.0.11.2  netmask 0xffffff00
              nd6 options=21 <performnud,auto_linklocal>groups: tun openvpn
              Opened by PID 18216
      ovpnc2: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
              options=80000 <linkstate>inet6 fe80::12c3:7bff:fe47:e006%ovpnc2 prefixlen 64 scopeid 0x9
              inet zzz.zzz.zzz.205 --> zzz.zzz.zzz.193  netmask 0xffffffc0
              nd6 options=21 <performnud,auto_linklocal>groups: tun openvpn
              Opened by PID 66191
      ovpnc3: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
              options=80000 <linkstate>inet6 fe80::12c3:7bff:fe47:e006%ovpnc3 prefixlen 64 scopeid 0xa
              inet 172.22.233.131 --> 172.22.233.129  netmask 0xffffff80
              nd6 options=21 <performnud,auto_linklocal>groups: tun openvpn
              Opened by PID 97458</performnud,auto_linklocal></linkstate></up,pointopoint,running,multicast></performnud,auto_linklocal></linkstate></up,pointopoint,running,multicast></performnud,auto_linklocal></linkstate></up,pointopoint,running,multicast></promisc></performnud,auto_linklocal></up,running></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,simplex,multicast>
      

      And here's ps uxaww | grep openvpn, if needed:

      root    18216   0.0  0.2  20352  6204  -  Ss   15:56    0:00.02 /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf
      root    66191   0.0  0.2  20352  6648  -  Ss   15:58    0:02.68 /usr/local/sbin/openvpn --config /var/etc/openvpn/client2.conf
      root    97458   0.0  0.2  20352  6652  -  Ss   15:58    0:00.09 /usr/local/sbin/openvpn --config /var/etc/openvpn/client3.conf
      
      1 Reply Last reply Reply Quote 0
      • jimpJ
        jimp Rebel Alliance Developer Netgate
        last edited by

        How are you setting these routes?

        If you are setting manual routes for a VPN, you should never make them under static routes (System > Routing, Static Routes tab). If that worked before, it was only by luck or coincidence.

        VPN routes should either be placed in the VPN itself (using "remote network" entries) or make use of policy routing.

        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

        Need help fast? Netgate Global Support!

        Do not Chat/PM for help!

        1 Reply Last reply Reply Quote 0
        • F
          fogelholk
          last edited by

          @jimp:

          How are you setting these routes?

          If you are setting manual routes for a VPN, you should never make them under static routes (System > Routing, Static Routes tab). If that worked before, it was only by luck or coincidence.

          VPN routes should either be placed in the VPN itself (using "remote network" entries) or make use of policy routing.

          Indeed, I had static rules set up under System > Routing > Static routes, because that was the only was I was able to get it to work like I wanted (in 2.3.x) and also setting a few Firewall > Rules where certain traffic has specific gateways set under Advanced options.

          I've now removed/disabled the Static Routes (under System > Routing) and added those CIDR ranges to the VPN Clients Remote Network on the pfSense.

          Seems to be working for now, I'll try to simulate ISP disconnects and see if it still works from there, otherwise I'll reply to this topic again.

          Thanks for the suggestion on setting the routes correctly :)

          1 Reply Last reply Reply Quote 0
          • F
            fogelholk
            last edited by

            The error seems to have arrived again, I had hoped moving the static routes to the correct place would have solved all routing issues I have, but something more must be tinkered with it seems.

            Here's what happens (and works for a couple of days, before something around OpenVPN connections/ISP disconnects occurs):

            On a road warrior I have set up all traffic that does not belong to my network (192.168.11.0/24), should use a specific gateway, which is through the VPN provider AzireVPN. This works perfectly fine for a couple of days, and after a few OpenVPN connects/disconnects it just completely stops sending traffic from the OpenVPN server on the pfsense (which the road warrior connects to) to the OpenVPN client set up on the pfsense (towards the VPN provider AzireVPN), until I do a complete restart of the pfsense.

            See attached image openvpn-server-rules.png for reference.

            If I change the Gateway to "Default" instead, which uses my ISPs ordinary connection, it works. The same issue occurs with the redacted line, which is a VPN connection from the pfsense to another place, which just have some specific networks routed through it (which jimp helped me move the specific routes for in the last post).

            Here's the updated netstat -rn, if needed:

            Internet:
            Destination        Gateway            Flags     Netif Expire
            default            xxx.xxx.51.1       UGS        igb1
            10.0.11.1          link#8             UHS         lo0
            10.0.11.2          link#8             UH       ovpns1
            tt.tt.ttt.145      xxx.xxx.51.1       UGHS       igb1
            yy.yyy.y.20/31     xxx.xxx.51.1       UGS        igb1
            127.0.0.1          link#3             UH          lo0
            172.22.233.0/25    172.22.233.1       UGS      ovpnc3
            172.22.233.1       link#10            UH       ovpnc3
            172.22.233.3       link#10            UHS         lo0
            192.168.11.0/24    link#1             U          igb0
            192.168.11.1       link#1             UHS         lo0
            xxx.xxx.51.0/25    link#2             U          igb1
            xxx.xxx.51.94      link#2             UHS         lo0
            sss.sss.0.10       xxx.xxx.51.1       UGHS       igb1
            uuu.uuu.uuu.2      xxx.xxx.51.1       UGHS       igb1
            zzz.zzz.zzz.128/26 zzz.zzz.zzz.129    UGS      ovpnc2
            zzz.zzz.zzz.129    link#9             UH       ovpnc2
            zzz.zzz.zzz.139    link#9             UHS         lo0
            vvv.v.vv.231/32    xxx.xxx.51.1       UGS        igb1
            www.ww.ww.90       xxx.xxx.51.1       UGHS       igb1
            

            Not sure what more information is needed, but ask away if you need more! And again, this all works perfectly fine in pfSense 2.3.x, routes and gateway rules doesn't stop working after a few days.

            openvpn-server-rules.png
            openvpn-server-rules.png_thumb

            1 Reply Last reply Reply Quote 0
            • F
              fogelholk
              last edited by

              I'm still having this problem, but instead of troubleshooting the issues I'm having, maybe guiding me through how I should set up the following might help me get rid of my problems:

              I have pfSense set up at home which I have connected through a VPN provider which I want all but a few specific local IPs to use as their default gateway. I have currently set this up under System > Routing > Gateways where I have set the VPN provider interface/gateway as default. Under Firewall > Rules > LAN I have added a few IP addresses that uses my ISPs as the gateway instead.

              I have a VPN connection to my workplace, which only a few IP addresses on my network are allowed through by also setting up things under Firewall > Rules > LAN using IP aliases with lists of the clients on my local network allowed through the workplace VPN, and a few IP-ranges that should be routed through the VPN. This rule has a gateway set up which was also created under System > Routing > Gateways in a similair fasion as the VPN provider gateway.

              All of this seems to work without any problem as far as I can tell, I can surf the web with "all clients except a few specific local IPs" via the VPN provider, and I can reach my workplace from the specific clients from my network.

              The following is currently not working for me, it works for a few days if I restart pfSense until something changes (VPN provider reconnect or such):

              I have a openvpn server running on pfSense for roadwarrior purposes (phone, laptop and so on), I want my roadwarrior to default via my VPN provider, and also be able to reach to workplace via the VPN connection running from my pfSense. I have set rules under different tabs on Firewall > Rules, which works without trouble for a couple of days, then it just stops working until I restart pfSense.

              This is the only thing that breaks after a few days, when I'm home I can reach my workplace and surf via the VPN provider without any trouble.

              This all worked perfectly fine in pfSense 2.3.x.

              I hope this information can help someone guide me through the correct setting for such scenario, if I have misconfigured something.

              1 Reply Last reply Reply Quote 0
              • F
                fogelholk
                last edited by

                It seems like I have finally solved this while debugging some other issues I've had with an OpenVPN client connection.

                The final solution for this problem seems to have been that pfSense cannot set up routes correctly to openvpn client connections, and instead falls back to setting the interface as "lo0" when checking netstat -rn. I debugged this while creating a dummy route towards 1.2.3.4/32 under "Static routes" and attempted setting my OpenVPN clients gateway as the route, and then checked "netstat -rn" for the results.

                After editing my OpenVPN client setting " IPv4 Remote network(s)" and adding 0.0.0.0/0 to it, I am able to set this connection as gateway for certain addresses and, for example, DNS-servers under General Settings and seeing pfSense finally setting the gateway to "ovpnc2" as the interface instead of "lo0".

                1 Reply Last reply Reply Quote 0
                • First post
                  Last post
                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.