VPN up but routing problems (no traffic)
-
First of all you did not give any details about your set up in terms of subnets you are using. Please do. Secondly you should definitely start from rules at LAN interface unless you are pinging from the firewall itself.
And what does it mean:but the routing tables on site_1 pfSense show the client that pinged (pung?) from site_2. Although the gateway shows as link#5 as below:```
Destination Gateway Flags etc...
10.0.0.11 link#5 UHLW .... -
The subnets do not overlap.
site_1 = 192.168.10.0 /24
site_2 = 10.0.0.0 /24I'm not blocking anything on my LAN interface:
* LAN net * * * *The VPN is connected successfully according to Status:IPSEC.
When I ping from the site_2 -> site_1 I get no reply. BUT in the pfsense routing tables on site_1 I get an entry for the client that pinged from site_2 (in this case 10.0.0.11). However the entry for that client in the routing tables is wrong, the gateway is showing as link#5 (my WAN is link#6), and the interface is sk4 (my WAN interface is xl0).
EDIT
I should add that when the vpn is down, pinging from either site brings it up again. It's like site_1 knows site_2's subnet is across the VPN and enables it, but then goes ahead and routes any requests to that network through an entirely different interface.ANOTHER EDIT
I just noticed on site_1 that there is a route entry for 10.0.0.0 /8 that specifies the wrong gateway/interface (link#5/sk4). I manually deleted that entry, and added 10.0.0.0/24. Now the clients from site_2 show up in the routing table on site_1 with the interface xl0 (which I think is correct), but i'm still not getting any traffic through.The firewall logs in site_1 show the ping from site_2 clients being passed, so packets are definitely going through the VPN from site_2. But somehow, replies aren't coming back. Also, pings from site_1 to site_2 don't show at all on site_2's logs.
-
May we have from site1:
ifconfig netstat -rn setkey -D -P -
#ifconfig
sk0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=b <rxcsum,txcsum,vlan_mtu>ether 00:15:e9:41:4d:ec inet 192.168.10.1 netmask 0xffffff00 broadcast 192.168.10.255 inet6 fe80::215:e9ff:fe41:4dec%sk0 prefixlen 64 scopeid 0x1 media: Ethernet autoselect (1000baseTX <full-duplex>) status: active sk1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=b <rxcsum,txcsum,vlan_mtu>ether 00:15:e9:41:4d:75 inet 192.168.20.1 netmask 0xffffff00 broadcast 192.168.20.255 inet6 fe80::215:e9ff:fe41:4d75%sk1 prefixlen 64 scopeid 0x2 media: Ethernet autoselect (1000baseTX <full-duplex,flag0,flag1>) status: active sk2: flags=28943 <up,broadcast,running,promisc,simplex,multicast,ppromisc>metric 0 mtu 1500 options=b <rxcsum,txcsum,vlan_mtu>ether 00:11:95:f7:3e:6a inet 192.168.30.1 netmask 0xffffff00 broadcast 192.168.30.255 inet6 fe80::211:95ff:fef7:3e6a%sk2 prefixlen 64 scopeid 0x3 media: Ethernet autoselect (100baseTX <full-duplex,flag0,flag1>) status: active sk3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=b <rxcsum,txcsum,vlan_mtu>ether 00:1b:11:11:ba:6f inet 192.168.40.254 netmask 0xffffff00 broadcast 192.168.40.255 inet6 fe80::21b:11ff:fe11:ba6f%sk3 prefixlen 64 scopeid 0x4 media: Ethernet autoselect (100baseTX <full-duplex,flag0,flag1>) status: active sk4: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=b <rxcsum,txcsum,vlan_mtu>ether 00:1b:11:11:ba:6c inet 10.1.1.254 netmask 0xffffffff broadcast 10.1.1.254 inet6 fe80::21b:11ff:fe11:ba6c%sk4 prefixlen 64 scopeid 0x5 inet 10.1.2.254 netmask 0xff000000 broadcast 10.255.255.255 inet 10.1.3.254 netmask 0xff000000 broadcast 10.255.255.255 inet 10.1.4.254 netmask 0xff000000 broadcast 10.255.255.255 inet 10.1.5.1 netmask 0xff000000 broadcast 10.255.255.255 inet 192.168.1.254 netmask 0xffffff00 broadcast 192.168.1.255 inet 192.168.50.1 netmask 0xffffff00 broadcast 192.168.50.255 inet 192.168.0.2 netmask 0xffffff00 broadcast 192.168.0.255 media: Ethernet autoselect (100baseTX <full-duplex,flag0,flag1>) status: active xl0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=9 <rxcsum,vlan_mtu>ether 00:b0:d0:e4:ad:05 inet 192.168.11.2 netmask 0xffffff00 broadcast 192.168.11.255 inet6 fe80::2b0:d0ff:fee4:ad05%xl0 prefixlen 64 scopeid 0x6 media: Ethernet autoselect (100baseTX <full-duplex>) status: active plip0: flags=108810 <pointopoint,simplex,multicast,needsgiant>metric 0 mtu 1500 lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8 enc0: flags=41 <up,running>metric 0 mtu 1536 pflog0: flags=100 <promisc>metric 0 mtu 33204 pfsync0: flags=41 <up,running>metric 0 mtu 1460 pfsync: syncdev: lo0 syncpeer: 224.0.0.240 maxupd: 128</up,running></promisc></up,running></up,loopback,running,multicast></pointopoint,simplex,multicast,needsgiant></full-duplex></rxcsum,vlan_mtu></up,broadcast,running,simplex,multicast></full-duplex,flag0,flag1></rxcsum,txcsum,vlan_mtu></up,broadcast,running,simplex,multicast></full-duplex,flag0,flag1></rxcsum,txcsum,vlan_mtu></up,broadcast,running,simplex,multicast></full-duplex,flag0,flag1></rxcsum,txcsum,vlan_mtu></up,broadcast,running,promisc,simplex,multicast,ppromisc></full-duplex,flag0,flag1></rxcsum,txcsum,vlan_mtu></up,broadcast,running,simplex,multicast></full-duplex></rxcsum,txcsum,vlan_mtu></up,broadcast,running,simplex,multicast>#netstat -rn
Routing tables Internet: Destination Gateway Flags Refs Use Netif Expire default 192.168.11.1 UGS 0 505947 xl0 10.0.0.0/24 link#6 UCS 0 0 xl0 10.0.0.10 link#6 UHLW 1 26 xl0 10.1.1.0/24 link#5 UCS 0 0 sk4 10.1.1.11 00:11:24:cf:d7:6c UHLW 1 0 sk4 558 10.1.1.254/32 link#5 UC 0 0 sk4 10.1.2.0/24 link#5 UCS 0 0 sk4 10.1.2.10 00:16:cb:a2:db:ea UHLW 1 10 sk4 687 10.1.3.0/24 link#5 UCS 0 0 sk4 10.1.3.10 00:11:24:76:af:b2 UHLW 1 0 sk4 179 10.1.4.0/24 link#5 UCS 0 0 sk4 10.1.4.10 00:16:cb:aa:7a:4c UHLW 1 0 sk4 640 10.1.5.0/24 link#5 UCS 0 0 sk4 10.10.15.0/24 192.168.10.254 UGS 0 698434 sk0 127.0.0.1 127.0.0.1 UH 0 0 lo0 192.168.0.0/24 link#5 UC 0 0 sk4 192.168.0.52 00:16:cb:a4:00:32 UHLW 1 0 sk4 942 192.168.1.0/24 link#5 UC 0 0 sk4 192.168.10.0/24 link#1 UC 0 0 sk0 192.168.10.10 00:11:09:24:7b:61 UHLW 1 8740 sk0 1181 192.168.10.11 00:08:a1:3c:72:5e UHLW 1 16311 sk0 1111 192.168.10.12 00:19:e3:e7:f5:2e UHLW 1 11013 sk0 887 192.168.10.15 00:0c:29:07:48:61 UHLW 1 1 sk0 855 192.168.10.17 00:0c:29:74:22:a3 UHLW 1 2737 sk0 925 192.168.10.18 00:0c:29:86:8f:2d UHLW 1 1010 sk0 985 192.168.10.19 00:1f:f3:c6:95:f9 UHLW 1 1 sk0 485 192.168.10.32 00:00:21:2d:14:a4 UHLW 1 27006 sk0 1113 192.168.10.34 00:1f:f3:cd:b6:55 UHLW 1 87 sk0 848 192.168.10.41 00:19:e3:49:b5:1e UHLW 1 10 sk0 1094 192.168.10.42 00:0c:76:ee:6d:7d UHLW 1 3392 sk0 871 192.168.10.45 00:0c:76:1b:e5:a3 UHLW 1 25353 sk0 1098 192.168.10.46 00:1f:5b:ea:a2:f6 UHLW 1 1 sk0 861 192.168.10.47 00:22:41:25:13:62 UHLW 1 43 sk0 889 192.168.10.48 00:0d:93:b4:52:24 UHLW 1 1 sk0 969 192.168.10.51 00:0c:29:81:88:c6 UHLW 1 6836 sk0 960 192.168.10.57 00:1e:52:f2:ef:99 UHLW 1 1 sk0 888 192.168.10.58 00:1e:c2:1d:3e:d5 UHLW 1 2 sk0 232 192.168.10.60 00:16:cb:a2:81:09 UHLW 1 2 sk0 938 192.168.10.64 00:1e:c2:1e:42:71 UHLW 1 250 sk0 449 192.168.10.68 00:1e:c2:13:3d:15 UHLW 1 41 sk0 902 192.168.10.100 00:15:f2:6d:98:c5 UHLW 1 12399 sk0 660 192.168.10.108 00:12:3f:45:79:2a UHLW 1 11377 sk0 1020 192.168.10.113 00:0e:0c:75:ec:72 UHLW 1 0 sk0 818 192.168.10.121 00:17:f2:00:ba:17 UHLW 1 250 sk0 865 192.168.10.122 00:17:f2:0b:ea:b8 UHLW 1 8 sk0 850 192.168.10.123 00:17:f2:0b:ea:64 UHLW 1 8 sk0 853 192.168.10.124 00:0d:93:4d:ad:1c UHLW 1 76 sk0 570 192.168.10.126 00:0d:93:3f:ca:74 UHLW 1 42 sk0 762 192.168.10.129 00:1f:5b:ea:a1:1b UHLW 1 74 sk0 894 192.168.10.140 00:16:cb:88:b7:c4 UHLW 1 60 sk0 503 192.168.10.190 00:24:8c:37:99:c7 UHLW 1 9085 sk0 1017 192.168.10.191 00:16:cb:ab:1c:a9 UHLW 1 24 sk0 1186 192.168.10.193 00:21:e9:63:4d:54 UHLW 1 0 sk0 530 192.168.10.200 00:16:cb:a5:1a:0c UHLW 1 38 sk0 1161 192.168.10.230 00:17:f2:04:ed:02 UHLW 1 183 sk0 152 192.168.10.232 00:1e:c2:1e:3a:8c UHLW 1 1 sk0 836 192.168.10.234 00:1e:c2:a8:4c:40 UHLW 1 3 sk0 466 192.168.10.235 00:1f:5b:f6:be:5a UHLW 1 7 sk0 222 192.168.10.236 00:1f:5b:3f:b9:47 UHLW 1 16 sk0 918 192.168.10.237 00:1f:5b:e8:d0:c1 UHLW 1 1 sk0 840 192.168.10.240 00:1e:4f:c2:07:d4 UHLW 1 472 sk0 1194 192.168.10.254 00:21:27:c9:03:09 UHLW 1 25 sk0 887 192.168.11.0/24 link#6 UC 0 0 xl0 192.168.11.1 00:90:96:86:ec:81 UHLW 2 6125 xl0 996 192.168.20.0/24 link#2 UC 0 0 sk1 192.168.20.10 00:04:23:c3:2e:21 UHLW 1 19291130 sk1 1063 192.168.20.206 00:17:f2:0d:d8:29 UHLW 1 0 sk1 574 192.168.20.208 00:11:24:2e:03:78 UHLW 1 0 sk1 463 192.168.20.209 00:0d:93:56:7c:e8 UHLW 1 0 sk1 1004 192.168.20.211 00:0d:93:af:ce:5a UHLW 1 0 sk1 1164 192.168.30.0/24 link#3 UC 0 0 sk2 192.168.30.10 00:17:95:14:3d:c0 UHLW 1 0 sk2 897 192.168.40.0/24 link#4 UC 0 0 sk3 192.168.50.0/24 link#5 UC 0 0 sk4 Internet6: Destination Gateway Flags Netif Expire ::1 ::1 UHL lo0 fe80::%sk0/64 link#1 UC sk0 fe80::215:e9ff:fe41:4dec%sk0 00:15:e9:41:4d:ec UHL lo0 fe80::%sk1/64 link#2 UC sk1 fe80::215:e9ff:fe41:4d75%sk1 00:15:e9:41:4d:75 UHL lo0 fe80::%sk2/64 link#3 UC sk2 fe80::211:95ff:fef7:3e6a%sk2 00:11:95:f7:3e:6a UHL lo0 fe80::%sk3/64 link#4 UC sk3 fe80::21b:11ff:fe11:ba6f%sk3 00:1b:11:11:ba:6f UHL lo0 fe80::%sk4/64 link#5 UC sk4 fe80::21b:11ff:fe11:ba6c%sk4 00:1b:11:11:ba:6c UHL lo0 fe80::%xl0/64 link#6 UC xl0 fe80::2b0:d0ff:fee4:ad05%xl0 00:b0:d0:e4:ad:05 UHL lo0 fe80::%lo0/64 fe80::1%lo0 U lo0 fe80::1%lo0 link#8 UHL lo0 ff01:1::/32 link#1 UC sk0 ff01:2::/32 link#2 UC sk1 ff01:3::/32 link#3 UC sk2 ff01:4::/32 link#4 UC sk3 ff01:5::/32 link#5 UC sk4 ff01:6::/32 link#6 UC xl0 ff01:8::/32 ::1 UC lo0 ff02::%sk0/32 link#1 UC sk0 ff02::%sk1/32 link#2 UC sk1 ff02::%sk2/32 link#3 UC sk2 ff02::%sk3/32 link#4 UC sk3 ff02::%sk4/32 link#5 UC sk4 ff02::%xl0/32 link#6 UC xl0 ff02::%lo0/32 ::1 UC lo0#setkey -D -P
192.168.10.0/24[any] 192.168.10.1[any] any in none spid=5 seq=3 pid=28696 refcnt=1 10.0.0.0/24[any] 192.168.10.0/24[any] any in ipsec esp/tunnel/121.98.196.77-192.168.11.2/unique#16388 spid=8 seq=2 pid=28696 refcnt=1 192.168.10.1[any] 192.168.10.0/24[any] any out none spid=6 seq=1 pid=28696 refcnt=1 192.168.10.0/24[any] 10.0.0.0/24[any] any out ipsec esp/tunnel/192.168.11.2-121.98.196.77/unique#16387 spid=7 seq=0 pid=28696 refcnt=1The 10.0.0.0/24 link#6 in the netstat output is what I manually entered with 'route add', the 10.0.0.10 entry comes up when you either ping that client from site_1 or you ping from that client at site_2.
ifconfig shows sk4 with a bunch of aliases that I added to the firewall config with <shellcmd>. Initially I thought this was the problem, because this was all on the WAN interface, so I separated the WAN out to it's own interface but still no-go.
While collecting the ifconfig data I noticed that the aliases were broadcasting on 10.255.255.255. That seemed like a smoking gun so I altered the shellcmd entries, e.g.
<shellcmd>ifconfig sk4 10.1.1.254 netmask 255.255.255.255 alias</shellcmd>rebooted and now it's even worse ::) Pings still show up in the logs but no route entry shows up in the routing table. Here's the new ifconfig output:
sk0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=b <rxcsum,txcsum,vlan_mtu>ether 00:15:e9:41:4d:ec inet 192.168.10.1 netmask 0xffffff00 broadcast 192.168.10.255 inet6 fe80::215:e9ff:fe41:4dec%sk0 prefixlen 64 scopeid 0x1 media: Ethernet autoselect (1000baseTX <full-duplex,flag2>) status: active sk1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=b <rxcsum,txcsum,vlan_mtu>ether 00:15:e9:41:4d:75 inet 192.168.20.1 netmask 0xffffff00 broadcast 192.168.20.255 inet6 fe80::215:e9ff:fe41:4d75%sk1 prefixlen 64 scopeid 0x2 media: Ethernet autoselect (1000baseTX <full-duplex,flag0,flag1>) status: active sk2: flags=28943 <up,broadcast,running,promisc,simplex,multicast,ppromisc>metric 0 mtu 1500 options=b <rxcsum,txcsum,vlan_mtu>ether 00:11:95:f7:3e:6a inet 192.168.30.1 netmask 0xffffff00 broadcast 192.168.30.255 inet6 fe80::211:95ff:fef7:3e6a%sk2 prefixlen 64 scopeid 0x3 media: Ethernet autoselect (100baseTX <full-duplex,flag0,flag1>) status: active sk3: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=b <rxcsum,txcsum,vlan_mtu>ether 00:1b:11:11:ba:6f inet 192.168.40.254 netmask 0xffffff00 broadcast 192.168.40.255 inet6 fe80::21b:11ff:fe11:ba6f%sk3 prefixlen 64 scopeid 0x4 media: Ethernet autoselect (100baseTX <full-duplex,flag0,flag1>) status: active sk4: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=b <rxcsum,txcsum,vlan_mtu>ether 00:1b:11:11:ba:6c inet 10.1.1.254 netmask 0xffffffff broadcast 10.1.1.254 inet6 fe80::21b:11ff:fe11:ba6c%sk4 prefixlen 64 scopeid 0x5 inet 10.1.2.254 netmask 0xffffffff broadcast 10.1.2.254 inet 10.1.3.254 netmask 0xffffffff broadcast 10.1.3.254 inet 10.1.4.254 netmask 0xffffffff broadcast 10.1.4.254 inet 10.1.5.1 netmask 0xffffffff broadcast 10.1.5.1 inet 192.168.1.254 netmask 0xffffffff broadcast 192.168.1.254 inet 192.168.50.1 netmask 0xffffffff broadcast 192.168.50.1 inet 192.168.0.2 netmask 0xffffffff broadcast 192.168.0.2 media: Ethernet autoselect (100baseTX <full-duplex,flag0,flag1>) status: active xl0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500 options=9 <rxcsum,vlan_mtu>ether 00:b0:d0:e4:ad:05 inet 192.168.11.2 netmask 0xffffff00 broadcast 192.168.11.255 inet6 fe80::2b0:d0ff:fee4:ad05%xl0 prefixlen 64 scopeid 0x6 media: Ethernet autoselect (100baseTX <full-duplex>) status: active plip0: flags=108810 <pointopoint,simplex,multicast,needsgiant>metric 0 mtu 1500 lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384 inet 127.0.0.1 netmask 0xff000000 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x8 enc0: flags=41 <up,running>metric 0 mtu 1536 pflog0: flags=100 <promisc>metric 0 mtu 33204 pfsync0: flags=41 <up,running>metric 0 mtu 1460 pfsync: syncdev: lo0 syncpeer: 224.0.0.240 maxupd: 128</up,running></promisc></up,running></up,loopback,running,multicast></pointopoint,simplex,multicast,needsgiant></full-duplex></rxcsum,vlan_mtu></up,broadcast,running,simplex,multicast></full-duplex,flag0,flag1></rxcsum,txcsum,vlan_mtu></up,broadcast,running,simplex,multicast></full-duplex,flag0,flag1></rxcsum,txcsum,vlan_mtu></up,broadcast,running,simplex,multicast></full-duplex,flag0,flag1></rxcsum,txcsum,vlan_mtu></up,broadcast,running,promisc,simplex,multicast,ppromisc></full-duplex,flag0,flag1></rxcsum,txcsum,vlan_mtu></up,broadcast,running,simplex,multicast></full-duplex,flag2></rxcsum,txcsum,vlan_mtu></up,broadcast,running,simplex,multicast> ```</shellcmd> -
Briefly: it's a mess.
What is the reason you are using shellcmd? -
I'm using shellcmd to set up IP aliasing, which to my knowledge you can't do in the GUI. We need sk4 to be different IP's to different subnets. Basically we host a semi-public service to different clients within the same building, each client has their own network and subnet specific to them.
-
I agree that this is a mess. If you want to segregate the different clients in the building you need VLANs or separate physical interfaces. Setting up separate subnets on the same segment not only offers no security whatsoever, it complicates the configuration and leads to problems such as this. I would suggest you either invest in a managed switch (Looks like HP 1700-8 might suffice, and is < $90, or there's lots of good 10/100 gear on eBay for $cheap) and move to a VLAN-based configuration or discard all these superfluous subnets.
You were right about the subnet masks on the 10.x.x.x aliases, they were all /8s, so all traffic to 10.* would go out that physical interface, regardless of any routes added by IPsec. Now that you've set them to /32s, no traffic will go out those interfaces for the 10.x.x.x subnets you've assigned. If you want this to work at all, you probably want /24s for those.
If it's still not working, I'd get rid of all the aliases and then maybe we can troubleshoot and help you get a basic configuration working, and you can then go back to your mess if you like.
-
We need sk4 to be different IP's to different subnets.
Could you explain this in more details?
As I see you already have several interfaces with different subnets (I suspect different clients). You do not need any IP from these subnets to be presented on your WAN.
Agree with Ktims - first step is to get rid of all your VIPs provided via shellcmd. -
Yep will do. Fresh install, set up the VPN, when it's working add the VIP's and see if it breaks. I'll let you know how I get on.
I agree that we need a managed switch and VLANs or separate interfaces for each subnet (that's a lot of multi-port cards), but I'm kinda constrained by budget here…
-
I don't know what your arrangement with your clients is like, but I don't think it'd be hard to sell them on a $50 investment each for proper segregation from each other, then go out and buy an inexpensive 24 port switch and set this up properly.
To be perfectly honest though, for my own sanity I have spent my own money to put in eBayed HP or 3com gear to replace aging and cheap SOHO equipment at some of my client's when I've been in your situation. It's just not worth my time and frustration with that garbage to save the $40-50 it costs me to buy a 24 port proper 10/100 switch from eBay. Then I can be happy with gear I know doesn't suck and spend a lot less time diagnosing network issues, give them a more secure and functional setup, and everybody's happy. No idea what your time is worth on this project, but I suspect it will quickly add up to more than that trying to get your setup working - and what you're trying to do is vastly inferior. It should be possible to make it work though…
-
I've set up two clean pfsense boxes in vmware.
Both 1.2.3-RC3.
Set up IPSec tunnel between them.
Firewall IPSEC rules are allow all on both boxes.
Triple checked the IPSEC rules on both boxes to make sure they're on identical settings.tunnel gets established when a client on either network pings the remote network. Both sites IPSEC logs show succesful connections. However clients on Site 1 can ping clients on Site 2, but Site 2 clients can only ping Site 1's pfsense address. Pings to clients on Site 1 time out.
Under diagnostics in pfsense, Site 2 shows entries for Site 1 network. But Site 1 shows no entries for Site 2.
Anyone have any ideas? Like I said both these boxes are squeaky clean, nothing fancy going on. In fact, the vpn is the only thing configured on them. You'd think this would "just work".
-
In case this helps anyone else, I eventually got this working on the physical setup in my original post. I replaced our old DSL modem with a Linksys WAG54G2, mainly because it features VPN passthrough. After that everything instantly worked.