Traffic Won't Route Through Outgoing VPN
-
Hey there. I have an interesting issue. I use ExpressVPN and setup 4 outgoing clients that different subnets are routed through using the firewall rules. Up until recently this has worked great but then all of a sudden (maybe after I updated) I am having problems. The problem seems to be that even though the traffic gets routed through the appropriate rule on the firewall, it doesn't actually make it out the specified gateway (VPN gateway). So when I check whats my ip I get my real public ip every time.
Some of my subnets were only configured with the outbound NAT rules to go through the VPN because I didn't ever need to go around the VPN. The subnets that I use for typical browsing also have an outbound NAT for the regular gateway because I have aliases set up to go around the VPN for sites that block VPN's. The subnets WITHOUT the NAT rules for the regular WAN lost connection entirely. This pretty much tells me that even though the firewall rules SAY to go out the VPN gateway, they don't.
The connections on the OpenVPN page all still show that they are up, and I have assigned IP's through all of them. I have already talked to ExpressVPN and they were not able to figure anything out.
Something else I have noticed that I find peculiar, is it seems that all of my DNS requests are being routed from the DMZ vpn IP. So in other words, the DNS requests ARE going out the VPN pipe but only the DMZ own. If I'm in the "Internal subnet" network, and I make a DNS request, it somehow gets piped out DMZ subnet VPN. More than that, there seems to be an extremely large amount of DNS traffic going out that VPN pipe. Now that COULD just be because maybe everything in my network is for some reason getting routed through it. HOWEVER, none of them are going to the right DNS addresses I have specified. It does not matter which addresses I specify, they keep going to the same ones.
If anybody could please help me with this that would be absolutely amazing. I am at a loss and it's such a weird complicated issue that came up AFTER everything was working great it just makes me want to scrap the whole network and dump pfsense.
ALSO: Quick question, does anybody know an easy way to map traffic originating on Pfsense to which process is creating it?
EDIT/UPDATE: Nevermind the DNS issues. I figured out where I had my configuration incorrect on that one. I'm just looking for the issues with the VPN now. Thank you
-
@crackerjackshot said in Traffic Won't Route Through Outgoing VPN:
maybe after I updated
Maybe or definitely? There were changes to OpenVPN in 2.5.x that dropped some weak ciphers and broke some people's configs. To debug this you will have to post more details starting with your LAN rules & outbound NATs to see if anything is weird that somehow used to work. Any errors in your OpenVPN log?
-
@kom Only maybe because I have been so busy I hadn't been using anything on my servers for a little while so I would not have noticed if they had lost connectivity immediately. And I updated just recently and then AFTER realized it was broken. Not sure if broken came first or if update came first. What would be the most pertinent information to throw up there? LAN rules are just pretty standard for any of the VPN config. I set up an all to all allow at the bottom of my LAN subnet and I put the default gateway as the VPN gateway created. Then the outbound NAT's are just setup to where they NAT the internal IP address to the VPN gateway.
Getting this in the OpenVPN logs. Not sure what that is about. I'll give any and all information that is relevant but I'm kind of not sure exactly what else to include.
-
@crackerjackshot Those are normal. It's pfSense checking on the status of the OpenVPN service. You're sure your account is paid up?
Can you ping the other end of the tunnel? If you run a packet capture on your OpenVPN interface, does it show any traffic?
-
@kom great questions. Im away from my place at the moment but ill check those out. I also found a weird error saying one of the gateways was "invalid". So ill show a screenshot of that as well. But despite that I know that all traffic other than the DNS on the DMZ VPN connection are bypassing it.
-
@kom Ok. Just checked and I can ping the servers on the other end. I did a packet capture on the outgoing VPN interface and got this...
Which is kinda bizarre because the VPN IP on the other side is 104.140.21.19. So what that IP is I don't even know that those ICMP echo requests are going to.
This is the warning I'm getting in the logs. I am not sure what this is about. I don't think it's even relevant to what is going on but I figured I'd include it. I don't think its relevant because its the only one getting that alarm but all of them are going around the VPN not just the one for internal.
And yes my account is paid up lol. I paid a year in advance just 2 months ago.
So did a DNS leak test to double check the DNS traffic and it is all properly being shown as coming out the VPN tunnel. Anything that isn't DNS, all the traffic being routed by the firewall rules through the VPN gateways is not. The DNS I don't think makes it to those rules because it gets handled from the DNS resolver.
-
@kom Just found this in the routing section... so I guess that is what this IP is.
-
@crackerjackshot Those 10.17.x.x are the IPs assigned to each endpoint of the tunnel. I don't know how you filtered but you should also be seeing ICMP echo replies, not just requests. It's time to post your LAN rules and outgoing NAT rules.
-
@kom I did not filter anything. I was trying to get as much traffic as possible. I was also able to get on the DMZ VPN interface and see that the DNS requests were making it out. Now there is some new weird info as well. Maybe it's not weird to you but I found some potential fixes to try when searching some more. One was to set the gateway to do not monitor it, and that just broke the internet entirely. The second was UNcheck the "do not pull routes" options, and once again... that broke the internet it seems.
These are the outgoing NAT rules and the Firewall rules for the Internal network. From now on I'll just post stuff specific to that network for trying to get this figured out. I'm assuming that whatever is wrong with it will be the same thing wrong with the others.
Would any of this be caused or exacerbated by the fact that I have all of my DNS requests getting piped out the one DMZ VPN connection in the DNS resolver? It worked before...
EDIT: Let me know if more detailed view into any of those rules is better.
EDIT2:
Just the traffic graph showing the connections on there. I would guess that this is just further showing that those pipes are only getting the echo requests out.
-
@crackerjackshot That all looks good to me, and like you've said it used to work. You said you could ping the other end of the tunnel but the trace only shows the requests and not the replies. Your graphs appear to show only tunnel overhead traffic. Can a client on the 10.100.2.0 network ping 8.8.8.8?
-
@kom Yes it can. I have full internet on that subnet. I'm even at work right now (though I've accidentally cut it off a couple of times trying some fixes) on that subnet. The only problem being that it is being routed around the VPN tunnel.
These two screenshots show the firewall logging the first one showing that it's being let out through the rule that should be doing the VPN pipe. Normally the second rule is the VPN address as the source, but it shows the let out anything from firewall host itself and then src is my real IP.
And there is an example of the DNS being routed out the VPN tunnel. And I've double checked that with DNS leak tests and seen that the IP is coming from the VPN, but the site knows my IP because of the 443 traffic.
-
@crackerjackshot Is it possible that's it's routing properly but you're just misinterpreting the result? If your DNS leak test site is showing your VPN IP address then it's working fine. If you go here which IP does it show, WAN or VPN?
https://whatismyipaddress.com/
Also, it's hard to see what you're doing with tiny snippets of rules or logs with all context stripped away. Obscure any public details like WAN address etc but otherwise more is better.
-
@kom said in Traffic Won't Route Through Outgoing VPN:
https://whatismyipaddress.com/
WAN, no I'm used to how it's supposed to look. What I was saying is that the DNS queries are showing the VPN IP but on any site I do a "whats my ip" I get my WAN IP. So DNS is going through VPN but not the 443 traffic. It's really weird.
Yeah ok, I'll do that in the future. I was just trying to keep it quick and easy to do at first.
-
@crackerjackshot You have Resolver's outgoing network interface set to the VPN?
-
@kom Correct.
-
@crackerjackshot said in Traffic Won't Route Through Outgoing VPN:
Just checked and I can ping the servers on the other end. I did a packet capture on the outgoing VPN interface and got this...
Which is kinda bizarre because the VPN IP on the other side is 104.140.21.19. So what that IP is I don't even know that those ICMP echo requests are going to.
This is the warning I'm getting in the logs. I am not sure what this is about. I don't think it's even relevant to what is going on but I figured I'd include it. I don't think its relevant because its the only one getting that alarm but all of them are going around the VPN not just the one for internal.
What you see in the packet capture is the gateway monitoring. pfSense sends ping requests to the servers VPN IP, but the other side does not response.
Hence pfSense will mark the gateway as down and skip the filter rule which are using this gateway. Therefore your traffic goes out to WAN.What the traffic graph is showing are just the outgoing ping requests, not even 70 bytes p.s.
If your VPN is connected fine, you can just disable the gateway monitoring in the VPN gateway settings or even better enter a public IP which is responding to pings.
To avoid pfSense from skipping gateway rules if it is down you can set a check at System > Advanced > Miscellaneous > Skip rules when gateway is down.
-
@viragomann Thank you. Right, thats what I had gathered about the outbound traffic on the graph. Those Gateways had always shown as down even though the traffic worked just fine. Maybe something in the update changed that. As far as I can tell the VPN's are all up and running just fine. I will try that here soon when I'm off work. I've already dumped my network enough for one work day haha.
Thank you! I'll let you know how it goes.
-
@viragomann Holy crap that finally did it. Thank you!!
Now one last thing that I'd like some input on that maybe you know what this is. I have been noticing lately that every once in a while a get these massive dumps of DNS requests that look like they just originate from the firewall itself.
That isn't normal is it?
-
@crackerjackshot
This is a completely other source IP as the OpenVPN clients IP shown in the capture above. If it is not your VPN interface IP something is wrong in your outbound NAT or these DNS requests are accidentally bypassed the resolver and have no outbound NAT rule.Anyway, if this is not your clients internal VPN IP the DNS requests won't succeed, hence the client tries again. This explains why there are clusters of requests.
-
@viragomann That IP is the address of the DMZ VPN connection, that is why it's different than the internal VPN connection IP. As I said in earlier parts of this thread I have all of the DNS piped out just the one DMZ connection. It seems to be working as my DNS gets resolved but that could be why that is happening. I can switch the resolver to go out "all" and see if that changes it.
