WG not routing or sending traffic
-
Thanks for the input but that didn't work either.
I had already selected the Wireguard interface as the default route in Routing. Single in/out and all trafic uses it.
I hadn't selected WG for the LAN FW Rule but as I'd explicitly named the WG Int as the default route I didn't think I needed to.
Either way it didn't work though.
G
-
Just created an account to ask you... did anything work out? I'm also having similar issues as your— no traffic, no nothing flowing through Wireguard interface when trying to setup WG client from a commercial VPN provider.
-
@wolfant said in WG not routing or sending traffic:
Just created an account to ask you... did anything work out? I'm also having similar issues as your— no traffic, no nothing flowing through Wireguard interface when trying to setup WG client from a commercial VPN provider.
Hi,
No I never did and I got no further with it. I did suspect it might have been due to some funky routing clashes as my home subnet (172.17.10) overlapped with the range my ISP was using to assign the tunnel with but due to the wonders of VMWare a quick re-numbering to 10. didn't make any difference. My ISP couldn't help either.
As there doesn't seem to be any way to definitively prove a tunnel is up I can't even tell if the tunnel is working. If I could at least I'd know then that it was a rules/NAT/routing issue. As it stands for all I know it might just be a "bug" with the implementation and my ISP.
I will have another go at it when I get the time but I need to find a good source that explains things as it's still a little confusing.
G
-
As there doesn't seem to be any way to definitively prove a tunnel is up I can't even tell if the tunnel is working.
Check Lawrence Systems' video at 8:38 https://youtu(dot)be/PinVqihuvBQ?t=518 when he ran 'wg show' command in pfSense shell. His output had "latest handshake," "transfer," and other such info in the output. Whereas mine (and perhaps yours as well) didn't have any such info. So I've assumed the handshake with the peer never takes place.
Perhaps there is some sort of bug present in Wireguard implementation of pfSense at the moment. Let's hope best for the future.
-
@wolfant The handshake works provided you have all the settings setup correctly. Paste screenshots of the following
- Wireguard config on pf
- WireGuard Peer config on pf
- WAN rule
- WG interface rule
- NAT rules
Heres some documentation to get you started -
https://docs.netgate.com/pfsense/en/latest/recipes/wireguard-ra.htmlEdit: wg show does not show me handshake," "transfer," and other such info in the output for me. I rely on the client side logs for this. For iPhone , Mac , the WG client has logs where you can see 'live' what's happening.
-
Interestingly i'm connected right now over a WG tunnel and my
wg show
doesn't say anything about latest handshake.Anyway I digress. I've had a few issues with WG tunnels myself, and I find the easiest way to start is to do a tcpdump in an appropriate interface and look for the UDP 51280 or whatever traffic. If it's going out, and not coming back, it's probably a key mismatch.
If its not going out, either your AllowedIP's, NAT rules or policy routing rules are probably the issue.
-
Hi,
OK left replies for a few days and I've had another go at this. All I will say is it's VERY flaky for me.
I upgraded my main firewall to 2.5 and that completely killed OpenVPN (that's a completely different issue). Nothing changed, nothing touched apart from the upgrade and nothing I did would allow me to re-connect the OpenVPN connection. Go back to a snapshot of 2.4.5 works fine. Anyway...
Based on some of the posts I just bit the bullet and went for a clean install. Ran the wizzard, assinged the interfaces, all working.
Set up Wireguard and nothing else. All works through the VPN. Great I think it's something to do with my 2.4.5 config. Save out the working config and reboot.
After a reboot nothing works again.
Reload in the config from the known working config. Fails and doesn't connect.
Change the port on the tunnel from 2049 to 2050 and it's working again.
Save config again.
Reboot and not working again.
Re loading the working config doesn't fix it.
Add "Any" "any" rules all over the place. Starts working at some point though no certainty it's a rule I've added (as it worked fine without those rules previously)At various times I've tcpdumped the interface and I see traffic going out but seemingly never coming back (but it's definitely working and I'm definitely connected through the VPN)
So here I am with a kind of working config. I'm dreading rebooting in case it stops but I'm going to have to give it a go.
@ab5g said in WG not routing or sending traffic:
@wolfant The handshake works provided you have all the settings setup correctly. Paste screenshots of the following
- Wireguard config on pf
- WireGuard Peer config on pf
- WAN rule
- WG interface rule
- NAT rules
Heres some documentation to get you started -
https://docs.netgate.com/pfsense/en/latest/recipes/wireguard-ra.htmlEdit: wg show does not show me handshake," "transfer," and other such info in the output for me. I rely on the client side logs for this. For iPhone , Mac , the WG client has logs where you can see 'live' what's happening.
I can't see what's happening with the tunnel as the other end is my ISP and pfs doesn't show any connection info.
Perhaps you'd be so kind though as to explain something to me I've never really understood.
You have physical and virtual interfaces under "Interfaces" and you have "Interfaces" under FW rules and NAT.
I (think!!) I understand that when you apply a FW rule you imagine standing on the interface and the rule applies to the inbound traffic on that interface.
I also think I understand that "Wireguard" is an interface group that contains one or more tunnels. You can apply FW/NAT rules either to the tunnel, so it only affects that specific tunnel, or you can apply it to the group and it affects all the tunnels.
What I'm not clear on at all is how a packet traverses from my PC to the VPN and what order things are applied in. Lets say I'm connecting from my PC to www.bbc.co.uk
I'm assuming it goes
PC > FW Lan Interface > NAT > FW VPN Interface > WAN Interface > ISP Router > VPN Provider > www.bbc.co.uk
But how are rules ordered and applied?
What IP's does each step see? What rules are applied at each stage? Lets give some examples
PC = 192.168.1.10
FW LAN Interface = 192.168.1.20
NAT = LAN Subnet (192.168.1.0/24 > 172.16.0.20/32)
Wireguard Interface = 172.16.0.20
WAN Interface = 192.168.2.20
ISP Router = 192.23.142.2
VPN Provider Wireguard Client = 81.92.202.114
Destination Address = www.bbc.co.ukSo how does that all work together? While I can understand the logical path it takes I can't get my head around how you'd write a rule chain to allow that traffic out and back. For example what rules are needed on the WAN interface? It clearly needs something as it's the carrier for creating the VPN tunnel but what is it seeing?
I just don't understand how each of these things interacts and I've tried reading easily 100 sites and nothing seems to explain this interaction. Maybe they don't because I don't need to know of course but they don't explain that either.
If anyone can give me any help with this I'd appreciate it.
G
-
@xxgbhxx All valid questions - maybe this docuemntation would be of some help
https://docs.netgate.com/pfsense/en/latest/nat/process-order.htmlIt is missing the Wireguard rule processing order - but based on the tcpdump I can guess the following happens
PC = 192.168.1.10
FW LAN Interface = 192.168.1.20
NAT = LAN Subnet (192.168.1.0/24 > 172.16.0.20/32)
Wireguard Interface Local = 172.16.0.20/32
Wireguard Interface Remote (VPN provider) = 172.16.0.0/16 or whatever
WAN Interface = 70.1.2.3
VPN Provider Wireguard Client = 81.92.202.114
Destination Address = www.bbc.co.uk- the LAN rule gets processed first - the rule is set to it is set to allow any LAN to any, the packet passes through.
- Because you've setup a policy based rule, matching source 192.168.1.10 which sets the gateway to Wireguard GW (WG_v4)
- then the LAN NAT rule kicks in - it translates the 192.168.1.10 to 172.16.0.20 (translating LAN to the Wireguard tunnel IP).
- then the Wireguard peer - comes in. Wireguard looks at the destination (www.bbc.co.uk) and tries to see if there is a peer matching the rule. If any of the allowed IP's is 0.0.0.0/0 , the packet is sent on this tunnel. If no peer is found, the packet will get dropped here. Now this is where things get interesting - at this point the logs show nothing if the packet is dropped. I have seen this happen in my case and unable to find out why.
- Once a peer is found, the outbound firewall rules come in, which mostly is allow any any. Then the WAN NAT rule comes in which would NAT 172.16.0.20 to the WAN IP 70.1.2.3
I hope this makes sense
-
@xxgbhxx I also use IVPN as my provider and I have it working. Here are some screen shots of how I have my various settings configured.
The "Address" above is the IP address that IVPN will assign to you when you take the Public Key shown above and link it into your account at IVPN. Just copy and paste it in IVPN's WireGuard Key Management in your account settings and then note the IP address they give you.
The ip address you see above in Endpoint is the IP address for one of IVPN's wireguard servers. You can just copy and paste the FQDN from IVPN's website for the server you want to use. The public key will be found on IVPN's website for the server you want to use.
Assign the Wireguard VPN to an Interface as above
Set the MSS and enable the Wireguard VPN interface as above. I used this article as a guide on the MSS values https://www.netgate.com/blog/wireguard-in-pfsense-2-5-performance.html
The Gateway settings are above.
Create an Outbound NAT rule as above. The IP address is your network that you want to route through the VPN. If you have multiple subnets that you want to route through then create additional rules for them.
Lastly, create a firewall rule on whatever interface you want to route out through IVPN as above. In the example above I am routing one specific device 192.168.163.8 which is on my LAN out IVPN. So that rule is at Firewall/Rules/WAN.I basically fumbled my way into making this work. I am by no means any kind of expert on wireguard or pfSense. So if any of the guru's around here have any suggestions on improving what I've done I'd be very happy to hear it and continue learning.
I hope that helps!
-
They have released a setup guide yesterday - https://www.ivpn.net/setup/router/pfsense-wireguard/
-
@ab5g It does make sense and thank you so much for taking the time to explain.
@dma_pf what a great writeup thanks so much.@ab5g Certainly that page (which I have no idea how I missed it in my reading but sometimes it's wood for the trees...) does make things a little clearer but it does lead you to a very complicated order or interdependencies depending on your setup.
So I've had another go at this over the weekend and I finally succeeded. (@dma_pf as you can see I'd fumbled my way to this same config as well)
I went back and restarted the whole process again. I'll post here exactly how I go it working in case it helps someone else (same as @dma_pf though I'd written this before he posted). I am not saying this is either secure (I'm certain it can be locked down further) or needed (I still need to do testing to make 100% sure these are all actually needed) but it worked for me.
- I installed a fresh 2.5 from scratch and accepted all the defaults
- I installed only 2 interfaces, WAN and LAN on the command line then went through the formalities in the wizzard GUI
- I set up Wireguard under VPN. For my ISP I needed to:
- Obtain my allocated peer IP from my ISP (to do this I needed to generate my key pair in the Tunnel GUI, paste the public key to my ISP and get the IP back from them)
- Enter in the IP given to me by my ISP for my end of the tunnel into "address" (the port here is irrelevant as you never get incoming connections from your ISP)
- Set up my ISP peer endpoint - they will provide you with their public key, their peer IP and the port you connect out on
- Set allowed IP's to 0.0.0.0/0 (you can tie this down if you want to but this is what I did)
- Set Keep Alive to 30 (this seemed to be a good trade off)
- Go into Interfaces and assign wg0, name it and enable it
- Go into NAT, Outbound NAT, Manual, add both WAN and Wireguard IF with an any/any rule (you can remove any of the other rules or you can make this more secure by specifying the source)
- Go to FW Rules, LAN, new rule to end
- Source LAN net, any/any/any, advanced - Gateway = Wireguard VPN
- Go to FW Rules, Wireguard-IF, new rule
- Any/any/any/any default gw
- Routing - set WANGW as the default route
After this I had it working and I was sending traffic out via the VPN. I needed to make one further change to DNS configuration to ensure I wasn't leaking DNS but apart from that it's working.
Two things.
First as mentioned I need to go back through all the config and tighten things down. I know what I need to tighten down (like all the "any" rules) but right now I don't know what they're going to be (so for example what IP subnet source the wireguard interface sees for the rule - very quick and easy to check/work out but just need to check what it is)
Second upgrading my existing 2.4.5 to 2.5 and/or installing my 2.4.5 config backup onto a fresh 2.5 install broke both for Wireguard. I don't know what, how or why but all I know is WG didn't work. A fresh install with EXACTLY the same WG config worked but the exact same config on my existing FW didn't work.
EDIT
OK I've now been running this a few days and it's hardly stable. I definitely need to do yet more testing to figure out what I've done wrong or what's causing the instability. I have since added in my second WAN link back into the mix and I'm not sure if it's this that's messing everything up. I need to work on it a bit more when I get the time but there's definite progress.
Thank you all so much for adding to the post, your efforts are greatly appreciated.
G
-
@pfsenss said in WG not routing or sending traffic:
https://www.ivpn.net/setup/router/pfsense-wireguard/
Ah I hadn't seen that and a great find.
I will go through that now when I have a chance and make sure I've done it right.
Thanks again for spotting that.
G
-
Here's the latest update
OK I'm starting to get suspicious something is bugged.
If I go back to 2 interfaces (LAN and WAN) and set it up as I outline above it works and seems to work through reboots.
When I start to build on that config by adding another external gateway it stops working at some point (not sure when exactly yet but I will experiment further).
First "bug" is when you try and follow the guidance from iVPN. It tells you to set the MSS for the LAN interface to 1412 but you can't without fist assinging a static V6 address. This is because when you save the 1412 change, it fails claiming you get
"The Router Advertisements Server is active on this interface and it can be used only with a static IPv6 configuration. Please disable the Router Advertisements Server service on this interface first, then change the interface configuration."
Basically the default for RA is to have it turned on even if you've not actually assigned IPv6 to and interface. This means it errors if you assign a new value for MSS. So put in a false Static v6 ip (::1 works) and then you can get into DHCP RA and turn off RA. Back to interfaces and you can then change it to 1412.
Second but occurs some point after I add the second external gateway. Just adding the interface in VMWare, adding it under interfaces and then rebooting seemed to work fine. Then I made some DNS changes and it stops working. Nothing I then do works even if I back out those changes to the exact same spot where it was all working.
I can also see elsewhere other people are experiencing similar types of issues (such as PBR not working and traffic not working)
I will always look at my setup and config first before I start blaming anything else but at the moment I'm struggling to get anything that is even slightly stable and consistent even with a pretty simple config.
Going to keep working on it and updating this thread until I come up with the answer/solution.
Again thanks for those who've inputted so far.
G
-
@xxgbhxx said in WG not routing or sending traffic:
"The Router Advertisements Server is active on this interface and it can be used only with a static IPv6 configuration. Please disable the Router Advertisements Server service on this interface first, then change the interface configuration."
This is a bug, search the forums and you'll find it. The fix is:
-
Temporarily enable IPv6 on your LAN interface and set IPv6 Configuration Type to "Static IPv6", then assign it an IPv6 address ("::1" without quotes works with mask /128). Then Save.
-
Go to Services/DHCPv6 Server & RA/LAN/Router Advertisements then set Router Mode to "Disabled". Then Save.
-
Go back to the LAN interface and set IPv6 Configuration Type back to "None" then Save.
You should now be able to change the MSS value.
-
-
@dma_pf said in WG not routing or sending traffic:
@xxgbhxx said in WG not routing or sending traffic:
"The Router Advertisements Server is active on this interface and it can be used only with a static IPv6 configuration. Please disable the Router Advertisements Server service on this interface first, then change the interface configuration."
This is a bug, search the forums and you'll find it. The fix is:
-
Temporarily enable IPv6 on your LAN interface and set IPv6 Configuration Type to "Static IPv6", then assign it an IPv6 address ("::1" without quotes works with mask /128). Then Save.
-
Go to Services/DHCPv6 Server & RA/LAN/Router Advertisements then set Router Mode to "Disabled". Then Save.
-
Go back to the LAN interface and set IPv6 Configuration Type back to "None" then Save.
You should now be able to change the MSS value.
I did search I didn't find it.
I posted the same fix above (I'd worked it out by myself, go me!)
Thanks for the update though
G
-
-
@xxgbhxx Can I make the suggestion of posting screenshots of your port forward, outbound NAT, WAN, LAN & WG rules? I'll be happy to look at them and compare them with my set up which is working.
-
@dma_pf said in WG not routing or sending traffic:
@xxgbhxx Can I make the suggestion of posting screenshots of your port forward, outbound NAT, WAN, LAN & WG rules? I'll be happy to look at them and compare them with my set up which is working.
Thank you so much for your kind offer I will do so this weekend. With 2 interfaces it's "working" at the moment and has been for 72 hours so I need to snapshot that then make the chanages with the third interface until it breaks so I can get a comparison up.
G
-
@dma_pf
Ok I've spent an hour going through this this morning. What I have found is simply adding the third interface breaks it.As I mentioned, I've had it running 3 days with only 2 interfaces and it has continued to work without issue. This is what is shown in the various screenshots for the WORKING config. I will follow these with the changes that broke the config. (by the way there's no port forwarding as the remote is the VPN provider who never initiate and inbound connection)
Architecture Network Diagram
Working DNS
Working Gateway
Working LAN FW Rules
Working VPN Firewall Rule
Working WAN Firewall Rule
Working Outbound NAT
Working WG Tunnel Rule
Working WG Peer Rule
So this is where it has been for the past 72 hours. Everything has been working perfectly fine (though to be fair I only have a single machine testing this but ever time I've gone to check it's worked without a problem).
I then did the following
In VMWARE
Add new "physical" Interface on the network of my other Internet connection (VMX2)In PF
- Interfaces add new interface VMX2
- Interfaces OPT2
-- Description "CleanWAN"
-- Static IPv4
-- Address 192.168.2.52/24
-- Add a new gateway
-- Gateway Name "CleanWAN_GW"
-- Gateway IP 192.168.2.254
-- Gateway Description "Clean Internet Feed" - Enable Interface
Then "Apply Changes"
Without touching or doing anything else at this point I reboot to check everything is working and it no longer does.
NOT Working Gateway
NOT Working CleanWAN Interface
Absolutely nothing else in the setup has changed or has been touched at this point. Unless I'm missing something fundamental (which I might be!) there is absolutely no reason this should stop working at this point.
I can always instantly tell if it's stopped working because the dashboard shows unable to tell if there's an update
Another point to note is that it continues to work right up until the point I re-boot. So the changes I apply to enable VMX2 and the CleanWAN interface work until I reboot which I think is why I've been confused/unsure what's been causing it.
So where do I go from here?
G
-
@xxgbhxx Thanks for your last post. I'm a little bit confused so hopefully you can answer a couple of questions. So here they are:
-
In the Architecture Network Diagram you have ISP1 and ISP2. Are these 2 actually different ISP's with unique IP addresses out to the internet? Can you describe their connection out to the net (cable, fiber, wireless, etc.)?
-
In the Architecture Network Diagram are Vmx1 and Vmx2 both virtual adapters? Are they bound to the the same physical NIC? If so are they bound to the same port on the NIC? How many ports are on that NIC?
-
Is pfSense installed virtually and hosted by Vmware? Are both Vmx1 and Vmx2 devices configured within the pfSense virtual machine settings?
-
How are you planning to determine what needs to go out to either Vmx1 or Vmx2? (certain devices, particular destinations, failover, etc.)
-
-
-
Yes they are completely different ISP's on completely different equipment (actually two completely different companies) with completely unique IP's on the internet. They're both Fibre to the Cabinet (FTTC) connections that come into the house on copper. They are even completely different piece of copper.
-
Yes Vmx0, 1 and 2 are three completely different virtual adapters. They are bound to the same NIC but it's a quad port NIC so each virtual network is on it's own physical port. It's a direct 1 to 1 connection.
-
Yes pfSense is VMWare hosted (and I've been running it in this config for 10 years). Vmx1 and 2 are virtual NIC's assigned to this VM in VSphere. Those two NIC's are on two virtual switches in VMware. Each switch is assigned a different physical uplink. One uplink is plugged into one ISP router, the other uplink is plugged into the other router.
-
As you can see from the rule above, it's set to send all LAN traffic to that interface. For the CleanWAN interface/connection, within pfBlocker I create alias which maps the ASN's for Netflix and Amazon Prime to a single Alias called "Clear_Connect". Once that's done I then put a new PBR rule above that LAN one for all IP's in those ASN's to route on the clear interface. This is how I've been using PF for the past 10 years with OpenVPN on he same ISP.
G
-
-
@xxgbhxx You default gateway IPv4 cannot be WIREGUARD_IVPN_WGV4. Set it to WAN and reboot / restart the WireGuard tunnel. It should start working then
-
The VPN providers own guide says to set it to the WG interface as do many other guides.
Can't look past it was working with 2 interfaces set to that. Are you suggesting that adding that thirsd interface means I can't leave it set to the WG interface?
Either way I've just tried it and it made no difference so I don't think it's that.
Thanks for the suggestion though.
G
-
@xxgbhxx - The default gateway should is for your entire system, I'm comfortable in having it set to the WAN links and then use policy based routing for the source IP's that I want to use the tunnel.
With default gateway for the system set as 'WIREGUARD_IVPN_WGV4' - you are telling the system to use it for everything. When the tunnel goes down - it may render things unreachable. Anyways, I've seen more predictable performance when not seeing the default gateway to the 'WIREGUARD_IVPN_WGV4'.Also from the Netgate docs
"Before assigning the interface, make sure default gateway for the firewall is not set to Automatic or the firewall may end up using the wg interface as the default gateway, which is unlikely to be the desired outcome. "You can delete the WG interface, reboot once - then set the default GW to WAN and then add the WG interface. After this add a route in LAN to match source host that you want to send over the tunnel.
-
@ab5g said in WG not routing or sending traffic:
@xxgbhxx - The default gateway should is for your entire system, I'm comfortable in having it set to the WAN links and then use policy based routing for the source IP's that I want to use the tunnel.
With default gateway for the system set as 'WIREGUARD_IVPN_WGV4' - you are telling the system to use it for everything. When the tunnel goes down - it may render things unreachable. Anyways, I've seen more predictable performance when not seeing the default gateway to the 'WIREGUARD_IVPN_WGV4'.Also from the Netgate docs
"Before assigning the interface, make sure default gateway for the firewall is not set to Automatic or the firewall may end up using the wg interface as the default gateway, which is unlikely to be the desired outcome. "You can delete the WG interface, reboot once - then set the default GW to WAN and then add the WG interface. After this add a route in LAN to match source host that you want to send over the tunnel.
As I see it you have 2 choices
- EVERYTHING goes VPN and PBR stuff on WAN that doesn't
- EVERYTHING goes WAN and PBR stuff on VPN that doesn't
You're suggesting 2 I've configured 1.
Either way it shouldn't matter, both should work. Option 1 DID work when I only had 2 interfaces.
For my system everything DOES go down the VPN except for a small subset of streaming services that don't allow VPN. I've not set up that PBR yet (but it's what @dma_pf was asking in question 4 above). Right now I'm not worried about that PBR bit.
Anyway, I did as you suggested. I deleted the WG interface and rebooted. I set the default GW to WAN and then re-added the WG interface. I then re-added the WG FW rule any/any and I've left the same LAN rule as above that tells all LAN traffic to use the tunnel. That didn't work.
I don't think it's anything to do with the way I've done it. I just don't think it's working as intended. There's indications that there's something "wrong" with PF and WG as theres another thread on PBR with two interfaces not working as instended/expected.
Keep the suggestions coming happy to try anything at this point.
G
-
@xxgbhxx said in WG not routing or sending traffic:
As I see it you have 2 choices
- EVERYTHING goes VPN and PBR stuff on WAN that doesn't
- EVERYTHING goes WAN and PBR stuff on VPN that doesn't
I've always used option 2. I've got interfaces for 2 native networks, 4 vlans a site-to-site OpenVPN tunnel, and 3 Wireguard connections to IVPN. My Default Gateway IPv4 is set to WAN_DHCP and the Default Gateway IPv6 is set to None. I then policy route my traffic either out the WAN or through IVPN (Wireguard) in rules per each interface.
In PF
Interfaces add new interface VMX2
Interfaces OPT2
-- Description "CleanWAN"
-- Static IPv4
-- Address 192.168.2.52/24
-- Add a new gateway
-- Gateway Name "CleanWAN_GW"
-- Gateway IP 192.168.2.254
-- Gateway Description "Clean Internet Feed"
Enable InterfaceThen "Apply Changes"
Without touching or doing anything else at this point I reboot to check everything is working and it no longer does.It seems very strange that just adding an interface would make the others not work. I'm assuming you've tried to ping out to the internet (8.8.8.8 and google.com) with a Source Address of WAN, Wireguard_IVPN and CleanWAN? Do you any get responses?
-
@dma_pf said in WG not routing or sending traffic:
@xxgbhxx said in WG not routing or sending traffic:
As I see it you have 2 choices
- EVERYTHING goes VPN and PBR stuff on WAN that doesn't
- EVERYTHING goes WAN and PBR stuff on VPN that doesn't
I've always used option 2. I've got interfaces for 2 native networks, 4 vlans a site-to-site OpenVPN tunnel, and 3 Wireguard connections to IVPN. My Default Gateway IPv4 is set to WAN_DHCP and the Default Gateway IPv6 is set to None. I then policy route my traffic either out the WAN or through IVPN (Wireguard) in rules per each interface.
In PF
Interfaces add new interface VMX2
Interfaces OPT2
-- Description "CleanWAN"
-- Static IPv4
-- Address 192.168.2.52/24
-- Add a new gateway
-- Gateway Name "CleanWAN_GW"
-- Gateway IP 192.168.2.254
-- Gateway Description "Clean Internet Feed"
Enable InterfaceThen "Apply Changes"
Without touching or doing anything else at this point I reboot to check everything is working and it no longer does.It seems very strange that just adding an interface would make the others not work. I'm assuming you've tried to ping out to the internet (8.8.8.8 and google.com) with a Source Address of WAN, Wireguard_IVPN and CleanWAN? Do you any get responses?
It's just a consequence of my simple setup. 99% of my traffic goes via VPN so it makes sense to me that's the default.
Yes it's strange, hence why I'm so confused. Right now, in it's broken state when I go to a command prompt on the firewall
No Interface Specified
8.8.8.8 no response
1.1.1.1 no response
192.168.1.1 (Internet gateway) responds
google.com (doesn't resolve as the DNS is the other side of the WG connection)Specify WAN (192.168.1.52)
8.8.8.8 responds
1.1.1.1 responds
192.168.1.1 responds
google.com no response (can't resolve)Specify Wireguard_IVPN (172.26..)
8.8.8.8 no response
1.1.1.1 no response
192.168.1.1 responds
google.com no response (can't resolve)Specify CleanWAN (192.168.2.52)
8.8.8.8 responds
1.1.1.1 responds
google.com no response (can't resolve)Perhaps the interesting one for me is the fact that using wg0 as the source interface it gets a response from the WAN gateway. I wouldn't expect the tunnel to be able to see that IP as it's on a completely different subnet range. The firewall has to be rooting somehow but I'm not quite sure how/why.
But apart from that it's pretty much responding as I'd expect. DNS doesn't work as it's not contactable through the VPN. Bypassing the VPN means everything you'd expect to be contactable is contactable.
I'm really at a loss. I am looking at a number of the other posts on this forum though and it's clear that WG isn't probably production ready yet. There's a number of people seemingly having a lot of "odd" behaviour, especially with multiple connections. If I hadn't built from complete scratch I'd put it down to some legacy issue with my existing firewall but this is a brand new 2.5 build with absolutely zero other config on it before hand. You're absolutely right, it should break just by adding an interface and yet, here we are.
Thanks again for your effort it really is hugely appreciated.
G
-
@xxgbhxx said in WG not routing or sending traffic:
Specify Wireguard_IVPN (172.26..)
8.8.8.8 no response
1.1.1.1 no response
192.168.1.1 responds
google.com no response (can't resolve)Try changing the Endpoint Address of the IVPN Peer from gb2.wg.ivpn.net to its actual IP address of 185.59.221.225 and repeat this test.
-
-
@xxgbhxx Something is keeping the Wireguard_IVPN from getting to the IVPN server. The reason I suggested changing the Endpoint Address to the IP address from the FQDN is because your DNS is set to use the DNS servers of IVPN. So it would make sense that they would not connect to the FQDN as it would not be able to resolve the FQDN and therefore fail to make the connection. But that did not resolve the issue. So I would think at this point it would be a NAT or Firewall rule issue.
For the time being I think it would be helpful to try the following:
- Verify that the settings for the Wireguard Tunnel are as follows:
-
The Interface wg0 Address needs to match the IP address tied to your key in the Key Management section of your IVPN account. The CIDR mask has to be /32. Make sure that Public Key for the interface matches exactly the key you used in the Key Management section to generate the IP address in IVPN.
-
In the PEER section of the tunnel continue to use the IP address of 185.59.221.255. Make sure that the complete public key, x0BTRaxsdxAd58ZyU2YMX4bmuj+Eg+8/urT2F3Vs1n8= is entered exactly. The Peer Wireguard Address needs to be 172.16.0.1. Port 2049 and keep alive 25 are ok.
-
Save and test as above.
- If 1 above didn't fix it then I'd suggest:
-
Checking your Firewall Rules for the Wireguard_VPN interface. In my setup there are no rules...it is empty.
-
Checking your LAN Rules. For now I would create a rule as follows:
- Protocol: IPv4 Any
- Source, Port, Destination, Port: Any
- Gateway: Wireguard_IVPN
-
Make sure you place the rule high enough in your rule order so that packets hit it. I'd make sure the rule is set for logging for now.
- Modify your NAT Rules shown in the Working Outbound NAT picture you posted above. For now I would create a rule as follows:
-
WAN Rule
- Interface: WAN
- Source: 172.17.10.0/24 and Source Port: Blank
- Destination and Destination Port: Blank
- NAT Address: WAN Address
-
Wireguard_IVPN Rule
- Interface: Wireguard_IVPN
- Source: 172.17.10.0/24 and Source Port: Blank
- Destination and Destination Port: Blank
- NAT Address: Wireguard_IVPN Address
This will duplicate the NAT and Firewall rules in my set up. Test along the way.....I would suggest doing so from Diagnostics/Ping in the GUI. At this point the only thing that I can see so far that would be different would be that I use unbound for DNS, my Default Gateway is set to WAN and I don't use multi WAN.
The multi WAN has me a bit puzzled because I've just never worked with it. I can't think my way through it yet as to whether or not there many need to be some NAT rule for the 192.168.1.0/24 and 192.168.2.0/24 networks. But we may be getting ahead of ourselves on that. So for now, work on the above and let me know what happens.
Good luck!
-
@dma_pf said in WG not routing or sending traffic:
@xxgbhxx Something is keeping the Wireguard_IVPN from getting to the IVPN server. The reason I suggested changing the Endpoint Address to the IP address from the FQDN is because your DNS is set to use the DNS servers of IVPN. So it would make sense that they would not connect to the FQDN as it would not be able to resolve the FQDN and therefore fail to make the connection. But that did not resolve the issue. So I would think at this point it would be a NAT or Firewall rule issue.
Good luck!
Yeh I get that. I had tried that in one of my previous attempts but it's always good to try again.
The rules are exactly as I posted above and haven't been changed. I will read through your suggestion tomorrow when I'm on my main PC instead of my laptop.
We need to not lose sight that this worked perfectly until I added the third interface. If any of what you'd suggested was different it wouldn't have worked at all. I've even been through multiple reloads on the previous config without issues.
I tell you what I'll do over the next few days is build another FW and test that too. That way we can rule out that it's something in the original config thats causing it.
Thanks again for your help here.
G
-
@xxgbhxx I thought of 2 other things to check.
-
Look at the System Time in the dashboard of pfSense. You have all traffic locked down to only going out through IVPN. I had a similar configuration on a remote site years ago. It resulted in a chicken and the egg scenario. The OpenVpn tunnel to IVPN could not connect because the time stamps were so off from each other. As a result, pfSense could not get the correct time because it could not resolve the FQDN to get to a time server. This caused all sorts of issues. I fixed the problem by making sure that I had some time servers defined by their actual IP addresses. My System/General Setup/Localization/Timeservers looks like this: us.pool.ntp.org 129.6.15.27 129.6.15.28 129.6.15.29
-
I'm not sure if the configuration you have been testing on is on your production environment or if your had a second virtual machine set up. If it's the latter you might need to reboot the modems when switching between the pfsense virtual machines. See this: https://forum.netgate.com/topic/97622/how-to-configure-wan-with-static-ip
I tell you what I'll do over the next few days is build another FW and test that too. That way we can rule out that it's something in the original config thats causing it.
If you do startup a fresh VM, I'd suggest going slow. Get the LAN & WAN up and working. Then get the IVPN wireguard tunnel up, but leave the DNS to the default unbound and the default gateway to WAN. When IVPN is working correctly then add the 2nd interface and see what happens.
Try to leave pfSense as close to its default settings. Just build it one step at a time and see where/if it breaks by testing at each step.
-
-
@dma_pf said in WG not routing or sending traffic:
@xxgbhxx I thought of 2 other things to check.
-
Look at the System Time in the dashboard of pfSense. You have all traffic locked down to only going out through IVPN. I had a similar configuration on a remote site years ago. It resulted in a chicken and the egg scenario. The OpenVpn tunnel to IVPN could not connect because the time stamps were so off from each other. As a result, pfSense could not get the correct time because it could not resolve the FQDN to get to a time server. This caused all sorts of issues. I fixed the problem by making sure that I had some time servers defined by their actual IP addresses. My System/General Setup/Localization/Timeservers looks like this: us.pool.ntp.org 129.6.15.27 129.6.15.28 129.6.15.29
-
I'm not sure if the configuration you have been testing on is on your production environment or if your had a second virtual machine set up. If it's the latter you might need to reboot the modems when switching between the pfsense virtual machines. See this: https://forum.netgate.com/topic/97622/how-to-configure-wan-with-static-ip
I tell you what I'll do over the next few days is build another FW and test that too. That way we can rule out that it's something in the original config thats causing it.
If you do startup a fresh VM, I'd suggest going slow. Get the LAN & WAN up and working. Then get the IVPN wireguard tunnel up, but leave the DNS to the default unbound and the default gateway to WAN. When IVPN is working correctly then add the 2nd interface and see what happens.
Try to leave pfSense as close to its default settings. Just build it one step at a time and see where/if it breaks by testing at each step.
-
Time is correct to within a couple of seconds. You're usually allowed a minute or more grace with most crypto/PKI
-
This is a test server not my production server. I'd already "broken" the production server by doing an in-situ 2.5 upgrade which broke everything. That was the primary reason why I decided to start from scratch. Unless I'm misreading, that problem is related to bridge mode, which I don't use. As far as the ADSL modems are concerned these are just devices on the LAN.
-
I did go slow with this onealready. I take snapshots of the config after each change as long as it's still working. As I mentioned I already had a working config with two interfaces. I still have that snapshotted. I can load up that snapshot in seconds and it's all working again.
In fact lets do just that.
Here's the screenshots I've literally just taken of that same system with 2 interfaces before I add the third. It's all working exactly how you/others describe. I have DNS set to a single server 172.16.0.1 which is IVPN. I can use my connected PC exactly as I would expect. I can re-boot and it re-connects without an issue.
As I mentioned above, I know instantly when it's working as you get this massage on the front page.
If you like, I can go from this point and I can take screenshots every single time I make a change so you can see what changes I'm making and at what point it breaks?
G
-
-
Actually I just did a test.
Just ADDING the third network card in VMWare is enough to kill it.
So fully working as per picture above. Works fine through reboots etc. Go into VMWare, add a third interface to the PF machine. Gets instantly detected in PF. I then simply reboot and I get this
You can see the WG in now down. I've not actually configured the IF in Interfaces>Assignments (though it appears obviously). All I did was add the IF in VMWare and reboot.
SO...
Is this an incompatibility with VMWare or in the way it adds interfaces. About to go play some more.
G
-
We have progress....
The thought occurred to me if it was something that pf didn't like about me adding a "live" virtual adapter. So I powered down the fw, added the adapter with the fw powered down and powered back up.
Working
Added the Interface in Assignments
Working
Rebooted
Working
SOOOOOOOOOOOOOOOOOOOOOOOOOOOO.....
It seems so far that ALL my problems were to do with how/when I was adding in that third interface. If you add it in after you build with the system live it locks the system so hard nothing works.
I have no ideas why but I have to assume this is a bug.
I will keep testing to see if I can break it. If nothing else this whole process has given me a far better understanding of the firewall.
Thanks
G
-
@xxgbhxx Glad to hear things are moving along!
-
@dma_pf said in WG not routing or sending traffic:
@xxgbhxx Glad to hear things are moving along!
Getting there.
I had not realised how important that 1412 MSS change would be. What I was finding was that I could happily browse to some websites (e.g. ivpn.net) but I couldn't browse to others (www.bbc.co.uk). I then made the change to MSS and everything I've tested is now working flawlessly.
So now I'm going to continue building up the environment and start adding the additional layers of things like snort, pfblocker for my blocklists and so on and the conditional access for my streaming.
So far it's looking good.
Again can I say thank you @dma_pf for your patience in working on this. Sometimes you just need a sounding board to work through things and maybe this thread will help others as there's some good knowledge in it.
I am still debating if this should be raised as a bug or if it's normal behavior.
@jimp is this a bug with pf, FreeBSD, WG or is this expected behaviour? (you only need to read first and last post to get the gist though there's a post in the middle that has a network diagram too). I will raise a bug report if this isn't normal.
G
-
Been almost 2 weeks now so thought I'd make one last update to this thread.
I have everything working. Just a few headline PSA's from my perspective for people who just skim the rest of the posts
- Reducing the MSS on the LAN interface is vitally important as without it certain URL's would not work. I reduced it to 1412 (per my ISP guidance) and it's been perfect ever since
- You can't add a new network interface in VMWare while pfSense is running as it breaks the networking when you reboot pfSense
- pfSense seemingly has no easy way of monitoring what's going on with the WG tunnel including no easy way to see if the tunnel is actually up or being negotiated other than pinging the far side gateway
- I suspect pfSense' WG implementation is not yet completely stable as there's a lot of niggly issues being reported in this thread with "odd" behaviour
- It doesn't matter whether you use policy based routing to encrypt all and PBR the unencrypted (as I have mine) or use PBR to only encrypt certain traffic - both work
- I never did get any word back from Netgate on whether this is a bug or known/expected behaviour
Now it's actually working it's rock stable and hasn't dropped a connection yet (which is something OpenVPN was doing on a daily basis for me). It's clearly a little faster than OpenVPN but that might just be my imagination.
Again a huge thanks to @dma_pf and others for their contribution.
G
-
@xxgbhxx Thanks for the update! I was wondering how things worked out for you and I'm really glad it's all good now.
Here's a suggestion based on our conversation.
You might want to create an interface group to another server or 2 of IVPN's. It's really to do. Just duplicate all that you've done to create the interface you have already made to IVPN (interface, routing, NAT, firewall rules) but use a different IVPN server.
When it's all done bind the interfaces together in System/Routing/Gateway Groups then use that interface group in your rules to route your traffic out the interface group. This will keep your internet up if one of the servers you are connected to goes down for any reason.
I'd use a servers from different data center providers for each connection (datapacket, M247) to provide another level of fail safe protection. Maybe one across the channel? IVPN gives you up to 7 simultaneous connections.
(Bonus Tip: Ping to different servers and see where your fastest responses come from. In my case I get quicker responses, and consistently faster speed tests, from 2 servers that are about 200 miles from me, as opposed to one only 60 miles away. Don't assume that the closest servers are going provide the fastest services. It all depends on the routing out on the net).
-
@dma_pf said in WG not routing or sending traffic:
@xxgbhxx Thanks for the update! I was wondering how things worked out for you and I'm really glad it's all good now.
Here's a suggestion based on our conversation.
You might want to create an interface group to another server or 2 of IVPN's. It's really to do. Just duplicate all that you've done to create the interface you have already made to IVPN (interface, routing, NAT, firewall rules) but use a different IVPN server.
When it's all done bind the interfaces together in System/Routing/Gateway Groups then use that interface group in your rules to route your traffic out the interface group. This will keep your internet up if one of the servers you are connected to goes down for any reason.
I'd use a servers from different data center providers for each connection (datapacket, M247) to provide another level of fail safe protection. Maybe one across the channel? IVPN gives you up to 7 simultaneous connections.
(Bonus Tip: Ping to different servers and see where your fastest responses come from. In my case I get quicker responses, and consistently faster speed tests, from 2 servers that are about 200 miles from me, as opposed to one only 60 miles away. Don't assume that the closest servers are going provide the fastest services. It all depends on the routing out on the net).
That's exactly what I did last week. ;)
I also set up and configured Squid and Snort.
This weekend I do DNSBL in pfBlockerNG then I'm done for a while :)
Many thanks for the tip though :)
G
-
@xxgbhxx Just thought I'd do a very quick update.
It happened to me again today and I've finally nailed EXACTLY what the issue was/is and it turns out it was an already known issue with VMWare/PfSense (gee thanks Netgate).
The issue is with VMWares allocation of NIC's. In VMware when you add new nics they number them vmx0 vmx1 vmx2 and so on. When you add a new card for some completely inexplicable reason, VMWare numbers the NEW card vmx0 and then bumps up the interface numbers of all the other cards (so what WAS vmx0 becomes vmx1). This immediately breaks pfSense and pretty much means you have to re-do all your interfaces and firewalls.
SO
The moral here is add as many interfaces from day one as you ever expect to use and if you DO decide to any later on, make sure you fully prep for the impact (because remembering interface names/locations from 9 months ago is not easy!)
Thought I'd leave this here in case anyone has the same issue.