Trouble with Source-Based Routing
-
Hi all! Networking novice here with a bit of an interesting setup that I'm trying to get working.
I've got a small /28 subnet set up as a local vlan here at home. All traffic on this subnet is sent via wireguard to a hub server where it joins other smaller subnets in the same /24 space. Essentially I'm connecting all these subnets across different locations into a psuedo-LAN at the hub server so that all the devices in the /24 can talk to each other. On the hub side things are working fine.
Here at home, I was previously using an Edgerouter and ip tables to get the subnet traffic to flow over the wireguard tunnel while maintaining a normal WAN/LAN network for the rest of my home devices. I recently switched over to a pfSense box and am trying to get the same thing working again. pfSense obviously doesn't use iptables so I've been trying to translate things over to source-based routing.
I've been using pings and tcpdumps to figure out where things are getting hung up. So far, I've found the following is going on:
- ICMP requests from outside devices make it all the way through the wireguard tunnel to the intended device on the local /28 subnet. So, inbound routing from the wireguard tunnel to the vlan seems to be OK
- The ICMP replies get dropped between the local vlan and the wireguard tunnel. They never hit the wireguard interface which makes me think I need to do more to get source-based routing properly set up
I've got the wireguard interface set up as a gateway, and in the vlan firewall I've got the following rule:
which I figured should be enough to push all the traffic from the /28 subnet over to the wireguard tunnel. However, it doesn't seem to be doing that.The only other firewall rule I've got as part of this setup is on the wireguard interface, which is just a simple "pass everything" rule.
So being inexperienced with this kind of thing, I'm a bit stumped. Apologies if I'm not being super clear with how everything works, so feel free to ask for clarification.
Thanks in advance for any help!
-
More testing this morning. It appears that pfSense is totally ignoring my firewall rule on the 41 vlan. Any pings from devices on the local /28 are going out through the normal WAN interface and ignoring the rule that's supposed to be sending them via the wireguard gateway.
Anybody know what I'm doing wrong?
-
Can you post your routing table?
-
I'll do that when I get home.
I haven't added any routes and I've got the wireguard routing table set to "off" so that it doesn't add any either. My allowed IPs is set to 0.0.0.0/0, and when it was on, it added a 0.0.0.0 route to the top of the table, pushing everything over the wireguard tunnel which was not at all desired.
My current speculation is something basic with the wireguard tunnel isn't set up right - I can't ping the endpoints properly and I'm guessing that's what's preventing the routing from working properly. I added a reply on the wireguard thread asking about this.
-
Output of netstat -r:
Routing tables Internet: Destination Gateway Flags Netif Expire default (public gateway) UGS vtnet0 44.48.41.16/28 link#7 U vtnet1.4 <-- the /28 I want going through wireguard 44.48.41.17 link#7 UHS lo0 <-- the address of the router on that /28 (public subnet)/22 link#1 U vtnet0 (public ip)/22 link#1 UHS lo0 localhost link#3 UH lo0 192.168.0.0/24 link#9 U vtnet1.1 192.168.0.1 link#9 UHS lo0 192.168.1.0/24 link#2 U vtnet1 router link#2 UHS lo0 192.168.16.1 link#10 UH tunwg0 <-- this entry gets added when wireguard is running 192.168.200.0/24 link#8 U vtnet1.2 192.168.200.1 link#8 UHS lo0 <WAN dns stuff> Internet6: (ipv6 stuff here)So as you can see, the address 192.168.16.1 is set up as available via the wireguard tunnel, even though it's actually just the local IP of the wireguard interface. So that's definitely not helping things. There's also no 192.168.16.1 on the loopback interface either.
It's very possible that these issues are a result of the unofficial wireguard support. Or I'm just dumb and don't know how to set things up properly. One of those is more likely :D
-
And just some additional info - this was all that was needed on the old edgerouter to get this setup working. I guess there's something to be said about the simplicity of debian-based networking:
ip rule add from 44.48.41.16/28 table NET44 ip route add 192.168.16.0/31 dev wg0 table NET44 ip route add default via 192.168.16.0 table NET44 ip route add 44.48.41.16/28 dev switch0.41 table NET44 -
So more troubleshooting this evening. Here's the current hangup - still stuck on outbound routing of packets from the /28 to the wireguard interface/gateway. Here's an illustration that I've put together based on tcpdump tracing of the ICMP packets.
Wireguard Firewall Rules:
Vlan 41 Firewall Rules:
And finally, the current output of netstat -r:
Destination Gateway Flags Netif Expire default (WAN Gateway) UGS vtnet0 44.48.41.16/28 link#7 U vtnet1.4 44.48.41.17 link#7 UHS lo0 (ISP Subnet)/22 link#1 U vtnet0 (WAN Address) link#1 UHS lo0 localhost link#3 UH lo0 192.168.0.0/24 link#9 U vtnet1.1 192.168.0.1 link#9 UHS lo0 192.168.1.0/24 link#2 U vtnet1 router link#2 UHS lo0 192.168.2.0/24 192.168.2.2 UGS ovpns1 192.168.2.1 link#10 UHS lo0 192.168.2.2 link#10 UH ovpns1 192.168.16.1 link#11 UH tunwg0 192.168.200.0/24 link#8 U vtnet1.2 192.168.200.1 link#8 UHS lo0 (ISP DNS Server 1) UHS vtnet0 (ISP DNS Server 2) UHS vtnet0Hopefully that clears things up a bit. Or makes everything even more confusing. Sorry to keep asking what could be a bunch of dumb questions.