WG not routing or sending traffic
-
@dma_pf said in WG not routing or sending traffic:
@xxgbhxx Something is keeping the Wireguard_IVPN from getting to the IVPN server. The reason I suggested changing the Endpoint Address to the IP address from the FQDN is because your DNS is set to use the DNS servers of IVPN. So it would make sense that they would not connect to the FQDN as it would not be able to resolve the FQDN and therefore fail to make the connection. But that did not resolve the issue. So I would think at this point it would be a NAT or Firewall rule issue.
Good luck!
Yeh I get that. I had tried that in one of my previous attempts but it's always good to try again.
The rules are exactly as I posted above and haven't been changed. I will read through your suggestion tomorrow when I'm on my main PC instead of my laptop.
We need to not lose sight that this worked perfectly until I added the third interface. If any of what you'd suggested was different it wouldn't have worked at all. I've even been through multiple reloads on the previous config without issues.
I tell you what I'll do over the next few days is build another FW and test that too. That way we can rule out that it's something in the original config thats causing it.
Thanks again for your help here.
G
-
@xxgbhxx I thought of 2 other things to check.
-
Look at the System Time in the dashboard of pfSense. You have all traffic locked down to only going out through IVPN. I had a similar configuration on a remote site years ago. It resulted in a chicken and the egg scenario. The OpenVpn tunnel to IVPN could not connect because the time stamps were so off from each other. As a result, pfSense could not get the correct time because it could not resolve the FQDN to get to a time server. This caused all sorts of issues. I fixed the problem by making sure that I had some time servers defined by their actual IP addresses. My System/General Setup/Localization/Timeservers looks like this: us.pool.ntp.org 129.6.15.27 129.6.15.28 129.6.15.29
-
I'm not sure if the configuration you have been testing on is on your production environment or if your had a second virtual machine set up. If it's the latter you might need to reboot the modems when switching between the pfsense virtual machines. See this: https://forum.netgate.com/topic/97622/how-to-configure-wan-with-static-ip
I tell you what I'll do over the next few days is build another FW and test that too. That way we can rule out that it's something in the original config thats causing it.
If you do startup a fresh VM, I'd suggest going slow. Get the LAN & WAN up and working. Then get the IVPN wireguard tunnel up, but leave the DNS to the default unbound and the default gateway to WAN. When IVPN is working correctly then add the 2nd interface and see what happens.
Try to leave pfSense as close to its default settings. Just build it one step at a time and see where/if it breaks by testing at each step.
-
-
@dma_pf said in WG not routing or sending traffic:
@xxgbhxx I thought of 2 other things to check.
-
Look at the System Time in the dashboard of pfSense. You have all traffic locked down to only going out through IVPN. I had a similar configuration on a remote site years ago. It resulted in a chicken and the egg scenario. The OpenVpn tunnel to IVPN could not connect because the time stamps were so off from each other. As a result, pfSense could not get the correct time because it could not resolve the FQDN to get to a time server. This caused all sorts of issues. I fixed the problem by making sure that I had some time servers defined by their actual IP addresses. My System/General Setup/Localization/Timeservers looks like this: us.pool.ntp.org 129.6.15.27 129.6.15.28 129.6.15.29
-
I'm not sure if the configuration you have been testing on is on your production environment or if your had a second virtual machine set up. If it's the latter you might need to reboot the modems when switching between the pfsense virtual machines. See this: https://forum.netgate.com/topic/97622/how-to-configure-wan-with-static-ip
I tell you what I'll do over the next few days is build another FW and test that too. That way we can rule out that it's something in the original config thats causing it.
If you do startup a fresh VM, I'd suggest going slow. Get the LAN & WAN up and working. Then get the IVPN wireguard tunnel up, but leave the DNS to the default unbound and the default gateway to WAN. When IVPN is working correctly then add the 2nd interface and see what happens.
Try to leave pfSense as close to its default settings. Just build it one step at a time and see where/if it breaks by testing at each step.
-
Time is correct to within a couple of seconds. You're usually allowed a minute or more grace with most crypto/PKI
-
This is a test server not my production server. I'd already "broken" the production server by doing an in-situ 2.5 upgrade which broke everything. That was the primary reason why I decided to start from scratch. Unless I'm misreading, that problem is related to bridge mode, which I don't use. As far as the ADSL modems are concerned these are just devices on the LAN.
-
I did go slow with this onealready. I take snapshots of the config after each change as long as it's still working. As I mentioned I already had a working config with two interfaces. I still have that snapshotted. I can load up that snapshot in seconds and it's all working again.
In fact lets do just that.
Here's the screenshots I've literally just taken of that same system with 2 interfaces before I add the third. It's all working exactly how you/others describe. I have DNS set to a single server 172.16.0.1 which is IVPN. I can use my connected PC exactly as I would expect. I can re-boot and it re-connects without an issue.
As I mentioned above, I know instantly when it's working as you get this massage on the front page.
If you like, I can go from this point and I can take screenshots every single time I make a change so you can see what changes I'm making and at what point it breaks?
G
-
-
Actually I just did a test.
Just ADDING the third network card in VMWare is enough to kill it.
So fully working as per picture above. Works fine through reboots etc. Go into VMWare, add a third interface to the PF machine. Gets instantly detected in PF. I then simply reboot and I get this
You can see the WG in now down. I've not actually configured the IF in Interfaces>Assignments (though it appears obviously). All I did was add the IF in VMWare and reboot.
SO...
Is this an incompatibility with VMWare or in the way it adds interfaces. About to go play some more.
G
-
We have progress....
The thought occurred to me if it was something that pf didn't like about me adding a "live" virtual adapter. So I powered down the fw, added the adapter with the fw powered down and powered back up.
Working
Added the Interface in Assignments
Working
Rebooted
Working
SOOOOOOOOOOOOOOOOOOOOOOOOOOOO.....
It seems so far that ALL my problems were to do with how/when I was adding in that third interface. If you add it in after you build with the system live it locks the system so hard nothing works.
I have no ideas why but I have to assume this is a bug.
I will keep testing to see if I can break it. If nothing else this whole process has given me a far better understanding of the firewall.
Thanks
G
-
@xxgbhxx Glad to hear things are moving along!
-
@dma_pf said in WG not routing or sending traffic:
@xxgbhxx Glad to hear things are moving along!
Getting there.
I had not realised how important that 1412 MSS change would be. What I was finding was that I could happily browse to some websites (e.g. ivpn.net) but I couldn't browse to others (www.bbc.co.uk). I then made the change to MSS and everything I've tested is now working flawlessly.
So now I'm going to continue building up the environment and start adding the additional layers of things like snort, pfblocker for my blocklists and so on and the conditional access for my streaming.
So far it's looking good.
Again can I say thank you @dma_pf for your patience in working on this. Sometimes you just need a sounding board to work through things and maybe this thread will help others as there's some good knowledge in it.
I am still debating if this should be raised as a bug or if it's normal behavior.
@jimp is this a bug with pf, FreeBSD, WG or is this expected behaviour? (you only need to read first and last post to get the gist though there's a post in the middle that has a network diagram too). I will raise a bug report if this isn't normal.
G
-
Been almost 2 weeks now so thought I'd make one last update to this thread.
I have everything working. Just a few headline PSA's from my perspective for people who just skim the rest of the posts
- Reducing the MSS on the LAN interface is vitally important as without it certain URL's would not work. I reduced it to 1412 (per my ISP guidance) and it's been perfect ever since
- You can't add a new network interface in VMWare while pfSense is running as it breaks the networking when you reboot pfSense
- pfSense seemingly has no easy way of monitoring what's going on with the WG tunnel including no easy way to see if the tunnel is actually up or being negotiated other than pinging the far side gateway
- I suspect pfSense' WG implementation is not yet completely stable as there's a lot of niggly issues being reported in this thread with "odd" behaviour
- It doesn't matter whether you use policy based routing to encrypt all and PBR the unencrypted (as I have mine) or use PBR to only encrypt certain traffic - both work
- I never did get any word back from Netgate on whether this is a bug or known/expected behaviour
Now it's actually working it's rock stable and hasn't dropped a connection yet (which is something OpenVPN was doing on a daily basis for me). It's clearly a little faster than OpenVPN but that might just be my imagination.
Again a huge thanks to @dma_pf and others for their contribution.
G
-
@xxgbhxx Thanks for the update! I was wondering how things worked out for you and I'm really glad it's all good now.
Here's a suggestion based on our conversation.
You might want to create an interface group to another server or 2 of IVPN's. It's really to do. Just duplicate all that you've done to create the interface you have already made to IVPN (interface, routing, NAT, firewall rules) but use a different IVPN server.
When it's all done bind the interfaces together in System/Routing/Gateway Groups then use that interface group in your rules to route your traffic out the interface group. This will keep your internet up if one of the servers you are connected to goes down for any reason.
I'd use a servers from different data center providers for each connection (datapacket, M247) to provide another level of fail safe protection. Maybe one across the channel? IVPN gives you up to 7 simultaneous connections.
(Bonus Tip: Ping to different servers and see where your fastest responses come from. In my case I get quicker responses, and consistently faster speed tests, from 2 servers that are about 200 miles from me, as opposed to one only 60 miles away. Don't assume that the closest servers are going provide the fastest services. It all depends on the routing out on the net).
-
@dma_pf said in WG not routing or sending traffic:
@xxgbhxx Thanks for the update! I was wondering how things worked out for you and I'm really glad it's all good now.
Here's a suggestion based on our conversation.
You might want to create an interface group to another server or 2 of IVPN's. It's really to do. Just duplicate all that you've done to create the interface you have already made to IVPN (interface, routing, NAT, firewall rules) but use a different IVPN server.
When it's all done bind the interfaces together in System/Routing/Gateway Groups then use that interface group in your rules to route your traffic out the interface group. This will keep your internet up if one of the servers you are connected to goes down for any reason.
I'd use a servers from different data center providers for each connection (datapacket, M247) to provide another level of fail safe protection. Maybe one across the channel? IVPN gives you up to 7 simultaneous connections.
(Bonus Tip: Ping to different servers and see where your fastest responses come from. In my case I get quicker responses, and consistently faster speed tests, from 2 servers that are about 200 miles from me, as opposed to one only 60 miles away. Don't assume that the closest servers are going provide the fastest services. It all depends on the routing out on the net).
That's exactly what I did last week. ;)
I also set up and configured Squid and Snort.
This weekend I do DNSBL in pfBlockerNG then I'm done for a while :)
Many thanks for the tip though :)
G
-
@xxgbhxx Just thought I'd do a very quick update.
It happened to me again today and I've finally nailed EXACTLY what the issue was/is and it turns out it was an already known issue with VMWare/PfSense (gee thanks Netgate).
The issue is with VMWares allocation of NIC's. In VMware when you add new nics they number them vmx0 vmx1 vmx2 and so on. When you add a new card for some completely inexplicable reason, VMWare numbers the NEW card vmx0 and then bumps up the interface numbers of all the other cards (so what WAS vmx0 becomes vmx1). This immediately breaks pfSense and pretty much means you have to re-do all your interfaces and firewalls.
SO
The moral here is add as many interfaces from day one as you ever expect to use and if you DO decide to any later on, make sure you fully prep for the impact (because remembering interface names/locations from 9 months ago is not easy!)
Thought I'd leave this here in case anyone has the same issue.