Pfsense 2.4.x routes broken/weird after some time. Working on 2.3.x.



  • I've been reading some threads with what would seem like similar problems, but didn't feel like hijacking their threads if it would result in not being the same issue.

    So here is is. I have a few static routes set up with rules allowing traffic between different VPN connections.

    Whenever, what seems to be consistent, my ISP or the VPN provider I connect through has a random disconnect, my static routes changes from the VPN interface to the localhost interface by checking netstat -rn which results in my roadwarrior clients not being able to utilize the static routes I've set up.

    The scenario that is working in pfSense 2.3, even through random disconnects, is that my roadwarrior client (OpenVPN) can connect through VPN connections I have set up on my pfSense to my work place.
    In 2.4.x (currently 2.4.2) this connection also works perfectly fine until, for example, my ISP has a random disconnect. After this my roadwarrior client can not send traffic through the VPN tunnel I have set up from pfSense to my work place.

    I hope I haven't obfuscated too much for it to be readable, but if a dev wants the real information, I'd be happy to send it privately if needed. If you need more information, just ask away! I really want to solve this and keep using pfSense 2.4, and not downgrade to 2.3 or restart pfSense weekly to have my roadwarrior working.

    Here's netstat when everything is working on pfSense 2.4.2:

    Internet:
    Destination        Gateway            Flags     Netif Expire
    default            zzz.zzz.zzz.205    UGS         lo0
    10.0.11.0/24       10.0.11.1          UGS         lo0
    10.0.11.1          link#8             UHS         lo0
    10.0.11.2          link#8             UH       ovpns1
    tt.tt.ttt.145      xxx.xxx.51.1       UGHS       igb1
    yy.yyy.y.0/21      172.22.233.131     UGS      ovpnc3
    yy.yyy.y.20/31     xxx.xxx.51.1       UGS        igb1
    127.0.0.1          link#3             UH          lo0
    172.21.0.0/16      172.22.233.131     UGS      ovpnc3
    172.22.0.0/16      172.22.233.131     UGS      ovpnc3
    172.22.233.128/25  172.22.233.129     UGS      ovpnc3
    172.22.233.129     link#10            UH       ovpnc3
    172.22.233.131     link#10            UHS         lo0
    192.168.11.0/24    link#1             U          igb0
    192.168.11.1       link#1             UHS         lo0
    xxx.xxx.51.0/25    link#2             U          igb1
    xxx.xxx.51.94      link#2             UHS         lo0
    sss.sss.0.10       xxx.xxx.51.1       UGHS       igb1
    uuu.uuu.uuu.2      xxx.xxx.51.1       UGHS       igb1
    zzz.zzz.zzz.192/26 zzz.zzz.zzz.193    UGS      ovpnc2
    zzz.zzz.zzz.193    link#9             UH       ovpnc2
    zzz.zzz.zzz.205    link#9             UHS         lo0
    vvv.v.vv.0/23      172.22.233.131     UGS      ovpnc3
    vvv.v.vv.231/32    xxx.xxx.51.1       UGS        igb1
    www.ww.ww.90       xxx.xxx.51.1       UGHS       igb1
    

    And here's netstat when it's Not working on pfsense 2.4.2:

    Internet:
    Destination        Gateway            Flags     Netif Expire
    default            zzz.zzz.zzz.144    UGS         lo0
    10.0.11.0/24       10.0.11.1          UGS         lo0
    10.0.11.1          link#9             UHS         lo0
    10.0.11.2          link#9             UH       ovpns1
    tt.tt.ttt.145      xxx.xxx.51.1       UGHS       igb1
    yy.yyy.y.0/21      172.22.233.131     UGS         lo0
    yy.yyy.y.20/31     xxx.xxx.51.1       UGS        igb1
    127.0.0.1          link#3             UH          lo0
    172.21.0.0/16      172.22.233.131     UGS         lo0
    172.22.0.0/16      172.22.233.131     UGS         lo0
    172.22.233.128/25  172.22.233.129     UGS      ovpnc3
    172.22.233.129     link#11            UH       ovpnc3
    172.22.233.131     link#11            UHS         lo0
    192.168.11.0/24    link#1             U          igb0
    192.168.11.1       link#1             UHS         lo0
    xxx.xxx.51.0/25    link#2             U          igb1
    xxx.xxx.51.94      link#2             UHS         lo0
    sss.sss.0.10       xxx.xxx.51.1       UGHS       igb1
    uuu.uuu.uuu.2      xxx.xxx.51.1       UGHS       igb1
    zzz.zzz.zzz.128/26 zzz.zzz.zzz.129    UGS      ovpnc2
    zzz.zzz.zzz.129    link#10            UH       ovpnc2
    zzz.zzz.zzz.144    link#10            UHS         lo0
    vvv.v.vv.0/23      172.22.233.131     UGS         lo0
    vvv.v.vv.231/32    xxx.xxx.51.1       UGS        igb1
    www.ww.ww.90       xxx.xxx.51.1       UGHS       igb1
    

    And here's an ifconfig output, also onfuscated to hell:

    igb0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=6403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6>ether mm:mm:mm:mm:mm:06
            hwaddr mm:mm:mm:mm:mm:06
            inet6 fe80::12c3:7bff:fe47:e006%igb0 prefixlen 64 scopeid 0x1
            inet 192.168.11.1 netmask 0xffffff00 broadcast 192.168.11.255
            nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
    igb1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=6403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6>ether mm:mm:mm:mm:mm:07
            hwaddr mm:mm:mm:mm:mm:07
            inet6 fe80::12c3:7bff:fe47:e007%igb1 prefixlen 64 scopeid 0x2
            inet xxx.xxx.51.94 netmask 0xffffff80 broadcast xxx.xxx.51.127
            nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
    lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
            options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet6 ::1 prefixlen 128
            inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
            inet 127.0.0.1 netmask 0xff000000
            nd6 options=21 <performnud,auto_linklocal>groups: lo
    enc0: flags=41 <up,running>metric 0 mtu 1536
            nd6 options=21 <performnud,auto_linklocal>groups: enc
    pflog0: flags=100 <promisc>metric 0 mtu 33160
            groups: pflog
    pfsync0: flags=0<> metric 0 mtu 1500
            groups: pfsync
            syncpeer: 224.0.0.240 maxupd: 128 defer: on
            syncok: 1
    ovpns1: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
            options=80000 <linkstate>inet6 fe80::12c3:7bff:fe47:e006%ovpns1 prefixlen 64 scopeid 0x8
            inet 10.0.11.1 --> 10.0.11.2  netmask 0xffffff00
            nd6 options=21 <performnud,auto_linklocal>groups: tun openvpn
            Opened by PID 18216
    ovpnc2: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
            options=80000 <linkstate>inet6 fe80::12c3:7bff:fe47:e006%ovpnc2 prefixlen 64 scopeid 0x9
            inet zzz.zzz.zzz.205 --> zzz.zzz.zzz.193  netmask 0xffffffc0
            nd6 options=21 <performnud,auto_linklocal>groups: tun openvpn
            Opened by PID 66191
    ovpnc3: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
            options=80000 <linkstate>inet6 fe80::12c3:7bff:fe47:e006%ovpnc3 prefixlen 64 scopeid 0xa
            inet 172.22.233.131 --> 172.22.233.129  netmask 0xffffff80
            nd6 options=21 <performnud,auto_linklocal>groups: tun openvpn
            Opened by PID 97458</performnud,auto_linklocal></linkstate></up,pointopoint,running,multicast></performnud,auto_linklocal></linkstate></up,pointopoint,running,multicast></performnud,auto_linklocal></linkstate></up,pointopoint,running,multicast></promisc></performnud,auto_linklocal></up,running></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,simplex,multicast>
    

    And here's ps uxaww | grep openvpn, if needed:

    root    18216   0.0  0.2  20352  6204  -  Ss   15:56    0:00.02 /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf
    root    66191   0.0  0.2  20352  6648  -  Ss   15:58    0:02.68 /usr/local/sbin/openvpn --config /var/etc/openvpn/client2.conf
    root    97458   0.0  0.2  20352  6652  -  Ss   15:58    0:00.09 /usr/local/sbin/openvpn --config /var/etc/openvpn/client3.conf
    

  • Rebel Alliance Developer Netgate

    How are you setting these routes?

    If you are setting manual routes for a VPN, you should never make them under static routes (System > Routing, Static Routes tab). If that worked before, it was only by luck or coincidence.

    VPN routes should either be placed in the VPN itself (using "remote network" entries) or make use of policy routing.



  • @jimp:

    How are you setting these routes?

    If you are setting manual routes for a VPN, you should never make them under static routes (System > Routing, Static Routes tab). If that worked before, it was only by luck or coincidence.

    VPN routes should either be placed in the VPN itself (using "remote network" entries) or make use of policy routing.

    Indeed, I had static rules set up under System > Routing > Static routes, because that was the only was I was able to get it to work like I wanted (in 2.3.x) and also setting a few Firewall > Rules where certain traffic has specific gateways set under Advanced options.

    I've now removed/disabled the Static Routes (under System > Routing) and added those CIDR ranges to the VPN Clients Remote Network on the pfSense.

    Seems to be working for now, I'll try to simulate ISP disconnects and see if it still works from there, otherwise I'll reply to this topic again.

    Thanks for the suggestion on setting the routes correctly :)



  • The error seems to have arrived again, I had hoped moving the static routes to the correct place would have solved all routing issues I have, but something more must be tinkered with it seems.

    Here's what happens (and works for a couple of days, before something around OpenVPN connections/ISP disconnects occurs):

    On a road warrior I have set up all traffic that does not belong to my network (192.168.11.0/24), should use a specific gateway, which is through the VPN provider AzireVPN. This works perfectly fine for a couple of days, and after a few OpenVPN connects/disconnects it just completely stops sending traffic from the OpenVPN server on the pfsense (which the road warrior connects to) to the OpenVPN client set up on the pfsense (towards the VPN provider AzireVPN), until I do a complete restart of the pfsense.

    See attached image openvpn-server-rules.png for reference.

    If I change the Gateway to "Default" instead, which uses my ISPs ordinary connection, it works. The same issue occurs with the redacted line, which is a VPN connection from the pfsense to another place, which just have some specific networks routed through it (which jimp helped me move the specific routes for in the last post).

    Here's the updated netstat -rn, if needed:

    Internet:
    Destination        Gateway            Flags     Netif Expire
    default            xxx.xxx.51.1       UGS        igb1
    10.0.11.1          link#8             UHS         lo0
    10.0.11.2          link#8             UH       ovpns1
    tt.tt.ttt.145      xxx.xxx.51.1       UGHS       igb1
    yy.yyy.y.20/31     xxx.xxx.51.1       UGS        igb1
    127.0.0.1          link#3             UH          lo0
    172.22.233.0/25    172.22.233.1       UGS      ovpnc3
    172.22.233.1       link#10            UH       ovpnc3
    172.22.233.3       link#10            UHS         lo0
    192.168.11.0/24    link#1             U          igb0
    192.168.11.1       link#1             UHS         lo0
    xxx.xxx.51.0/25    link#2             U          igb1
    xxx.xxx.51.94      link#2             UHS         lo0
    sss.sss.0.10       xxx.xxx.51.1       UGHS       igb1
    uuu.uuu.uuu.2      xxx.xxx.51.1       UGHS       igb1
    zzz.zzz.zzz.128/26 zzz.zzz.zzz.129    UGS      ovpnc2
    zzz.zzz.zzz.129    link#9             UH       ovpnc2
    zzz.zzz.zzz.139    link#9             UHS         lo0
    vvv.v.vv.231/32    xxx.xxx.51.1       UGS        igb1
    www.ww.ww.90       xxx.xxx.51.1       UGHS       igb1
    

    Not sure what more information is needed, but ask away if you need more! And again, this all works perfectly fine in pfSense 2.3.x, routes and gateway rules doesn't stop working after a few days.




  • I'm still having this problem, but instead of troubleshooting the issues I'm having, maybe guiding me through how I should set up the following might help me get rid of my problems:

    I have pfSense set up at home which I have connected through a VPN provider which I want all but a few specific local IPs to use as their default gateway. I have currently set this up under System > Routing > Gateways where I have set the VPN provider interface/gateway as default. Under Firewall > Rules > LAN I have added a few IP addresses that uses my ISPs as the gateway instead.

    I have a VPN connection to my workplace, which only a few IP addresses on my network are allowed through by also setting up things under Firewall > Rules > LAN using IP aliases with lists of the clients on my local network allowed through the workplace VPN, and a few IP-ranges that should be routed through the VPN. This rule has a gateway set up which was also created under System > Routing > Gateways in a similair fasion as the VPN provider gateway.

    All of this seems to work without any problem as far as I can tell, I can surf the web with "all clients except a few specific local IPs" via the VPN provider, and I can reach my workplace from the specific clients from my network.

    The following is currently not working for me, it works for a few days if I restart pfSense until something changes (VPN provider reconnect or such):

    I have a openvpn server running on pfSense for roadwarrior purposes (phone, laptop and so on), I want my roadwarrior to default via my VPN provider, and also be able to reach to workplace via the VPN connection running from my pfSense. I have set rules under different tabs on Firewall > Rules, which works without trouble for a couple of days, then it just stops working until I restart pfSense.

    This is the only thing that breaks after a few days, when I'm home I can reach my workplace and surf via the VPN provider without any trouble.

    This all worked perfectly fine in pfSense 2.3.x.

    I hope this information can help someone guide me through the correct setting for such scenario, if I have misconfigured something.



  • It seems like I have finally solved this while debugging some other issues I've had with an OpenVPN client connection.

    The final solution for this problem seems to have been that pfSense cannot set up routes correctly to openvpn client connections, and instead falls back to setting the interface as "lo0" when checking netstat -rn. I debugged this while creating a dummy route towards 1.2.3.4/32 under "Static routes" and attempted setting my OpenVPN clients gateway as the route, and then checked "netstat -rn" for the results.

    After editing my OpenVPN client setting " IPv4 Remote network(s)" and adding 0.0.0.0/0 to it, I am able to set this connection as gateway for certain addresses and, for example, DNS-servers under General Settings and seeing pfSense finally setting the gateway to "ovpnc2" as the interface instead of "lo0".