Pfsense 2.4.x routes broken/weird after some time. Working on 2.3.x.
-
I've been reading some threads with what would seem like similar problems, but didn't feel like hijacking their threads if it would result in not being the same issue.
So here is is. I have a few static routes set up with rules allowing traffic between different VPN connections.
Whenever, what seems to be consistent, my ISP or the VPN provider I connect through has a random disconnect, my static routes changes from the VPN interface to the localhost interface by checking netstat -rn which results in my roadwarrior clients not being able to utilize the static routes I've set up.
The scenario that is working in pfSense 2.3, even through random disconnects, is that my roadwarrior client (OpenVPN) can connect through VPN connections I have set up on my pfSense to my work place.
In 2.4.x (currently 2.4.2) this connection also works perfectly fine until, for example, my ISP has a random disconnect. After this my roadwarrior client can not send traffic through the VPN tunnel I have set up from pfSense to my work place.I hope I haven't obfuscated too much for it to be readable, but if a dev wants the real information, I'd be happy to send it privately if needed. If you need more information, just ask away! I really want to solve this and keep using pfSense 2.4, and not downgrade to 2.3 or restart pfSense weekly to have my roadwarrior working.
Here's netstat when everything is working on pfSense 2.4.2:
Internet: Destination Gateway Flags Netif Expire default zzz.zzz.zzz.205 UGS lo0 10.0.11.0/24 10.0.11.1 UGS lo0 10.0.11.1 link#8 UHS lo0 10.0.11.2 link#8 UH ovpns1 tt.tt.ttt.145 xxx.xxx.51.1 UGHS igb1 yy.yyy.y.0/21 172.22.233.131 UGS ovpnc3 yy.yyy.y.20/31 xxx.xxx.51.1 UGS igb1 127.0.0.1 link#3 UH lo0 172.21.0.0/16 172.22.233.131 UGS ovpnc3 172.22.0.0/16 172.22.233.131 UGS ovpnc3 172.22.233.128/25 172.22.233.129 UGS ovpnc3 172.22.233.129 link#10 UH ovpnc3 172.22.233.131 link#10 UHS lo0 192.168.11.0/24 link#1 U igb0 192.168.11.1 link#1 UHS lo0 xxx.xxx.51.0/25 link#2 U igb1 xxx.xxx.51.94 link#2 UHS lo0 sss.sss.0.10 xxx.xxx.51.1 UGHS igb1 uuu.uuu.uuu.2 xxx.xxx.51.1 UGHS igb1 zzz.zzz.zzz.192/26 zzz.zzz.zzz.193 UGS ovpnc2 zzz.zzz.zzz.193 link#9 UH ovpnc2 zzz.zzz.zzz.205 link#9 UHS lo0 vvv.v.vv.0/23 172.22.233.131 UGS ovpnc3 vvv.v.vv.231/32 xxx.xxx.51.1 UGS igb1 www.ww.ww.90 xxx.xxx.51.1 UGHS igb1And here's netstat when it's Not working on pfsense 2.4.2:
Internet: Destination Gateway Flags Netif Expire default zzz.zzz.zzz.144 UGS lo0 10.0.11.0/24 10.0.11.1 UGS lo0 10.0.11.1 link#9 UHS lo0 10.0.11.2 link#9 UH ovpns1 tt.tt.ttt.145 xxx.xxx.51.1 UGHS igb1 yy.yyy.y.0/21 172.22.233.131 UGS lo0 yy.yyy.y.20/31 xxx.xxx.51.1 UGS igb1 127.0.0.1 link#3 UH lo0 172.21.0.0/16 172.22.233.131 UGS lo0 172.22.0.0/16 172.22.233.131 UGS lo0 172.22.233.128/25 172.22.233.129 UGS ovpnc3 172.22.233.129 link#11 UH ovpnc3 172.22.233.131 link#11 UHS lo0 192.168.11.0/24 link#1 U igb0 192.168.11.1 link#1 UHS lo0 xxx.xxx.51.0/25 link#2 U igb1 xxx.xxx.51.94 link#2 UHS lo0 sss.sss.0.10 xxx.xxx.51.1 UGHS igb1 uuu.uuu.uuu.2 xxx.xxx.51.1 UGHS igb1 zzz.zzz.zzz.128/26 zzz.zzz.zzz.129 UGS ovpnc2 zzz.zzz.zzz.129 link#10 UH ovpnc2 zzz.zzz.zzz.144 link#10 UHS lo0 vvv.v.vv.0/23 172.22.233.131 UGS lo0 vvv.v.vv.231/32 xxx.xxx.51.1 UGS igb1 www.ww.ww.90 xxx.xxx.51.1 UGHS igb1And here's an ifconfig output, also onfuscated to hell:
igb0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=6403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6>ether mm:mm:mm:mm:mm:06 hwaddr mm:mm:mm:mm:mm:06 inet6 fe80::12c3:7bff:fe47:e006%igb0 prefixlen 64 scopeid 0x1 inet 192.168.11.1 netmask 0xffffff00 broadcast 192.168.11.255 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>) status: active igb1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=6403bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6>ether mm:mm:mm:mm:mm:07 hwaddr mm:mm:mm:mm:mm:07 inet6 fe80::12c3:7bff:fe47:e007%igb1 prefixlen 64 scopeid 0x2 inet xxx.xxx.51.94 netmask 0xffffff80 broadcast xxx.xxx.51.127 nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>) status: active lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384 options=600003 <rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6>inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 <performnud,auto_linklocal>groups: lo enc0: flags=41 <up,running>metric 0 mtu 1536 nd6 options=21 <performnud,auto_linklocal>groups: enc pflog0: flags=100 <promisc>metric 0 mtu 33160 groups: pflog pfsync0: flags=0<> metric 0 mtu 1500 groups: pfsync syncpeer: 224.0.0.240 maxupd: 128 defer: on syncok: 1 ovpns1: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500 options=80000 <linkstate>inet6 fe80::12c3:7bff:fe47:e006%ovpns1 prefixlen 64 scopeid 0x8 inet 10.0.11.1 --> 10.0.11.2 netmask 0xffffff00 nd6 options=21 <performnud,auto_linklocal>groups: tun openvpn Opened by PID 18216 ovpnc2: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500 options=80000 <linkstate>inet6 fe80::12c3:7bff:fe47:e006%ovpnc2 prefixlen 64 scopeid 0x9 inet zzz.zzz.zzz.205 --> zzz.zzz.zzz.193 netmask 0xffffffc0 nd6 options=21 <performnud,auto_linklocal>groups: tun openvpn Opened by PID 66191 ovpnc3: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500 options=80000 <linkstate>inet6 fe80::12c3:7bff:fe47:e006%ovpnc3 prefixlen 64 scopeid 0xa inet 172.22.233.131 --> 172.22.233.129 netmask 0xffffff80 nd6 options=21 <performnud,auto_linklocal>groups: tun openvpn Opened by PID 97458</performnud,auto_linklocal></linkstate></up,pointopoint,running,multicast></performnud,auto_linklocal></linkstate></up,pointopoint,running,multicast></performnud,auto_linklocal></linkstate></up,pointopoint,running,multicast></promisc></performnud,auto_linklocal></up,running></performnud,auto_linklocal></rxcsum,txcsum,rxcsum_ipv6,txcsum_ipv6></up,loopback,running,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,tso4,tso6,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,simplex,multicast>And here's ps uxaww | grep openvpn, if needed:
root 18216 0.0 0.2 20352 6204 - Ss 15:56 0:00.02 /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf root 66191 0.0 0.2 20352 6648 - Ss 15:58 0:02.68 /usr/local/sbin/openvpn --config /var/etc/openvpn/client2.conf root 97458 0.0 0.2 20352 6652 - Ss 15:58 0:00.09 /usr/local/sbin/openvpn --config /var/etc/openvpn/client3.conf
-
How are you setting these routes?
If you are setting manual routes for a VPN, you should never make them under static routes (System > Routing, Static Routes tab). If that worked before, it was only by luck or coincidence.
VPN routes should either be placed in the VPN itself (using "remote network" entries) or make use of policy routing.
-
How are you setting these routes?
If you are setting manual routes for a VPN, you should never make them under static routes (System > Routing, Static Routes tab). If that worked before, it was only by luck or coincidence.
VPN routes should either be placed in the VPN itself (using "remote network" entries) or make use of policy routing.
Indeed, I had static rules set up under System > Routing > Static routes, because that was the only was I was able to get it to work like I wanted (in 2.3.x) and also setting a few Firewall > Rules where certain traffic has specific gateways set under Advanced options.
I've now removed/disabled the Static Routes (under System > Routing) and added those CIDR ranges to the VPN Clients Remote Network on the pfSense.
Seems to be working for now, I'll try to simulate ISP disconnects and see if it still works from there, otherwise I'll reply to this topic again.
Thanks for the suggestion on setting the routes correctly :)
-
The error seems to have arrived again, I had hoped moving the static routes to the correct place would have solved all routing issues I have, but something more must be tinkered with it seems.
Here's what happens (and works for a couple of days, before something around OpenVPN connections/ISP disconnects occurs):
On a road warrior I have set up all traffic that does not belong to my network (192.168.11.0/24), should use a specific gateway, which is through the VPN provider AzireVPN. This works perfectly fine for a couple of days, and after a few OpenVPN connects/disconnects it just completely stops sending traffic from the OpenVPN server on the pfsense (which the road warrior connects to) to the OpenVPN client set up on the pfsense (towards the VPN provider AzireVPN), until I do a complete restart of the pfsense.
See attached image openvpn-server-rules.png for reference.
If I change the Gateway to "Default" instead, which uses my ISPs ordinary connection, it works. The same issue occurs with the redacted line, which is a VPN connection from the pfsense to another place, which just have some specific networks routed through it (which jimp helped me move the specific routes for in the last post).
Here's the updated netstat -rn, if needed:
Internet: Destination Gateway Flags Netif Expire default xxx.xxx.51.1 UGS igb1 10.0.11.1 link#8 UHS lo0 10.0.11.2 link#8 UH ovpns1 tt.tt.ttt.145 xxx.xxx.51.1 UGHS igb1 yy.yyy.y.20/31 xxx.xxx.51.1 UGS igb1 127.0.0.1 link#3 UH lo0 172.22.233.0/25 172.22.233.1 UGS ovpnc3 172.22.233.1 link#10 UH ovpnc3 172.22.233.3 link#10 UHS lo0 192.168.11.0/24 link#1 U igb0 192.168.11.1 link#1 UHS lo0 xxx.xxx.51.0/25 link#2 U igb1 xxx.xxx.51.94 link#2 UHS lo0 sss.sss.0.10 xxx.xxx.51.1 UGHS igb1 uuu.uuu.uuu.2 xxx.xxx.51.1 UGHS igb1 zzz.zzz.zzz.128/26 zzz.zzz.zzz.129 UGS ovpnc2 zzz.zzz.zzz.129 link#9 UH ovpnc2 zzz.zzz.zzz.139 link#9 UHS lo0 vvv.v.vv.231/32 xxx.xxx.51.1 UGS igb1 www.ww.ww.90 xxx.xxx.51.1 UGHS igb1Not sure what more information is needed, but ask away if you need more! And again, this all works perfectly fine in pfSense 2.3.x, routes and gateway rules doesn't stop working after a few days.
-
I'm still having this problem, but instead of troubleshooting the issues I'm having, maybe guiding me through how I should set up the following might help me get rid of my problems:
I have pfSense set up at home which I have connected through a VPN provider which I want all but a few specific local IPs to use as their default gateway. I have currently set this up under System > Routing > Gateways where I have set the VPN provider interface/gateway as default. Under Firewall > Rules > LAN I have added a few IP addresses that uses my ISPs as the gateway instead.
I have a VPN connection to my workplace, which only a few IP addresses on my network are allowed through by also setting up things under Firewall > Rules > LAN using IP aliases with lists of the clients on my local network allowed through the workplace VPN, and a few IP-ranges that should be routed through the VPN. This rule has a gateway set up which was also created under System > Routing > Gateways in a similair fasion as the VPN provider gateway.
All of this seems to work without any problem as far as I can tell, I can surf the web with "all clients except a few specific local IPs" via the VPN provider, and I can reach my workplace from the specific clients from my network.
The following is currently not working for me, it works for a few days if I restart pfSense until something changes (VPN provider reconnect or such):
I have a openvpn server running on pfSense for roadwarrior purposes (phone, laptop and so on), I want my roadwarrior to default via my VPN provider, and also be able to reach to workplace via the VPN connection running from my pfSense. I have set rules under different tabs on Firewall > Rules, which works without trouble for a couple of days, then it just stops working until I restart pfSense.
This is the only thing that breaks after a few days, when I'm home I can reach my workplace and surf via the VPN provider without any trouble.
This all worked perfectly fine in pfSense 2.3.x.
I hope this information can help someone guide me through the correct setting for such scenario, if I have misconfigured something.
-
It seems like I have finally solved this while debugging some other issues I've had with an OpenVPN client connection.
The final solution for this problem seems to have been that pfSense cannot set up routes correctly to openvpn client connections, and instead falls back to setting the interface as "lo0" when checking netstat -rn. I debugged this while creating a dummy route towards 1.2.3.4/32 under "Static routes" and attempted setting my OpenVPN clients gateway as the route, and then checked "netstat -rn" for the results.
After editing my OpenVPN client setting " IPv4 Remote network(s)" and adding 0.0.0.0/0 to it, I am able to set this connection as gateway for certain addresses and, for example, DNS-servers under General Settings and seeing pfSense finally setting the gateway to "ovpnc2" as the interface instead of "lo0".
