Traffic Won't Route Through Outgoing VPN
-
@kom great questions. Im away from my place at the moment but ill check those out. I also found a weird error saying one of the gateways was "invalid". So ill show a screenshot of that as well. But despite that I know that all traffic other than the DNS on the DMZ VPN connection are bypassing it.
-
@kom Ok. Just checked and I can ping the servers on the other end. I did a packet capture on the outgoing VPN interface and got this...
Which is kinda bizarre because the VPN IP on the other side is 104.140.21.19. So what that IP is I don't even know that those ICMP echo requests are going to.
This is the warning I'm getting in the logs. I am not sure what this is about. I don't think it's even relevant to what is going on but I figured I'd include it. I don't think its relevant because its the only one getting that alarm but all of them are going around the VPN not just the one for internal.
And yes my account is paid up lol. I paid a year in advance just 2 months ago.
So did a DNS leak test to double check the DNS traffic and it is all properly being shown as coming out the VPN tunnel. Anything that isn't DNS, all the traffic being routed by the firewall rules through the VPN gateways is not. The DNS I don't think makes it to those rules because it gets handled from the DNS resolver.
-
@kom Just found this in the routing section... so I guess that is what this IP is.
-
@crackerjackshot Those 10.17.x.x are the IPs assigned to each endpoint of the tunnel. I don't know how you filtered but you should also be seeing ICMP echo replies, not just requests. It's time to post your LAN rules and outgoing NAT rules.
-
@kom I did not filter anything. I was trying to get as much traffic as possible. I was also able to get on the DMZ VPN interface and see that the DNS requests were making it out. Now there is some new weird info as well. Maybe it's not weird to you but I found some potential fixes to try when searching some more. One was to set the gateway to do not monitor it, and that just broke the internet entirely. The second was UNcheck the "do not pull routes" options, and once again... that broke the internet it seems.
These are the outgoing NAT rules and the Firewall rules for the Internal network. From now on I'll just post stuff specific to that network for trying to get this figured out. I'm assuming that whatever is wrong with it will be the same thing wrong with the others.
Would any of this be caused or exacerbated by the fact that I have all of my DNS requests getting piped out the one DMZ VPN connection in the DNS resolver? It worked before...
EDIT: Let me know if more detailed view into any of those rules is better.
EDIT2:
Just the traffic graph showing the connections on there. I would guess that this is just further showing that those pipes are only getting the echo requests out.
-
@crackerjackshot That all looks good to me, and like you've said it used to work. You said you could ping the other end of the tunnel but the trace only shows the requests and not the replies. Your graphs appear to show only tunnel overhead traffic. Can a client on the 10.100.2.0 network ping 8.8.8.8?
-
@kom Yes it can. I have full internet on that subnet. I'm even at work right now (though I've accidentally cut it off a couple of times trying some fixes) on that subnet. The only problem being that it is being routed around the VPN tunnel.
These two screenshots show the firewall logging the first one showing that it's being let out through the rule that should be doing the VPN pipe. Normally the second rule is the VPN address as the source, but it shows the let out anything from firewall host itself and then src is my real IP.
And there is an example of the DNS being routed out the VPN tunnel. And I've double checked that with DNS leak tests and seen that the IP is coming from the VPN, but the site knows my IP because of the 443 traffic.
-
@crackerjackshot Is it possible that's it's routing properly but you're just misinterpreting the result? If your DNS leak test site is showing your VPN IP address then it's working fine. If you go here which IP does it show, WAN or VPN?
https://whatismyipaddress.com/
Also, it's hard to see what you're doing with tiny snippets of rules or logs with all context stripped away. Obscure any public details like WAN address etc but otherwise more is better.
-
@kom said in Traffic Won't Route Through Outgoing VPN:
https://whatismyipaddress.com/
WAN, no I'm used to how it's supposed to look. What I was saying is that the DNS queries are showing the VPN IP but on any site I do a "whats my ip" I get my WAN IP. So DNS is going through VPN but not the 443 traffic. It's really weird.
Yeah ok, I'll do that in the future. I was just trying to keep it quick and easy to do at first.
-
@crackerjackshot You have Resolver's outgoing network interface set to the VPN?
-
@kom Correct.
-
@crackerjackshot said in Traffic Won't Route Through Outgoing VPN:
Just checked and I can ping the servers on the other end. I did a packet capture on the outgoing VPN interface and got this...
Which is kinda bizarre because the VPN IP on the other side is 104.140.21.19. So what that IP is I don't even know that those ICMP echo requests are going to.
This is the warning I'm getting in the logs. I am not sure what this is about. I don't think it's even relevant to what is going on but I figured I'd include it. I don't think its relevant because its the only one getting that alarm but all of them are going around the VPN not just the one for internal.
What you see in the packet capture is the gateway monitoring. pfSense sends ping requests to the servers VPN IP, but the other side does not response.
Hence pfSense will mark the gateway as down and skip the filter rule which are using this gateway. Therefore your traffic goes out to WAN.What the traffic graph is showing are just the outgoing ping requests, not even 70 bytes p.s.
If your VPN is connected fine, you can just disable the gateway monitoring in the VPN gateway settings or even better enter a public IP which is responding to pings.
To avoid pfSense from skipping gateway rules if it is down you can set a check at System > Advanced > Miscellaneous > Skip rules when gateway is down.
-
@viragomann Thank you. Right, thats what I had gathered about the outbound traffic on the graph. Those Gateways had always shown as down even though the traffic worked just fine. Maybe something in the update changed that. As far as I can tell the VPN's are all up and running just fine. I will try that here soon when I'm off work. I've already dumped my network enough for one work day haha.
Thank you! I'll let you know how it goes.
-
@viragomann Holy crap that finally did it. Thank you!!
Now one last thing that I'd like some input on that maybe you know what this is. I have been noticing lately that every once in a while a get these massive dumps of DNS requests that look like they just originate from the firewall itself.
That isn't normal is it?
-
@crackerjackshot
This is a completely other source IP as the OpenVPN clients IP shown in the capture above. If it is not your VPN interface IP something is wrong in your outbound NAT or these DNS requests are accidentally bypassed the resolver and have no outbound NAT rule.Anyway, if this is not your clients internal VPN IP the DNS requests won't succeed, hence the client tries again. This explains why there are clusters of requests.
-
@viragomann That IP is the address of the DMZ VPN connection, that is why it's different than the internal VPN connection IP. As I said in earlier parts of this thread I have all of the DNS piped out just the one DMZ connection. It seems to be working as my DNS gets resolved but that could be why that is happening. I can switch the resolver to go out "all" and see if that changes it.
-
@crackerjackshot said in Traffic Won't Route Through Outgoing VPN:
That IP is the address of the DMZ VPN connection, that is why it's different than the internal VPN connection IP. As I said in earlier parts of this thread I have all of the DNS piped out just the one DMZ connection.
Well, but the outgoing interface the log is showing is "OUTGOINGEXPRESSVPNINT" which seems to me not being the DMZ interface as there is an other one you called "OUTGOINGEXPRESSVPNDMZ".
-
@viragomann Yeah that's interesting. I see what you're saying.
This log shows it more. All the internal traffic going out 443 displays the OUTGOINGEXPRESSVPNINT and it has the correct address. But the DNS traffic shows the same interface but then goes out the DMZ address. Up top theres a piece of traffic that shows the DMZ contacting the internet and it correctly shows it as OUTGOINGEXPRESSVPNDMZ.
Not sure what that's about
-
@crackerjackshot
So what is initiating these DNS requests?
I assume the DNS resolver on pfSense itself. If so is there any reason for using only the DMZ-VPN interface?
Check the resolvers outgoing interfaces and correct them if you want.Then configure your outbound NAT accordingly.
If it's the resolver and it should be able to go out on the INT-VPN add a rule to this interface for the source 127.0.0.0/8, if desired restrict it only for TCP/UDP 53. -
@viragomann You can see the DNS request just below the one going out the VPN pipe to the 1.1.1.1. It was originated on a machine in the internal net that has the 10.100.2.14 IP right now. Everything is set to query the .1 address in the subnet and then as far as my understanding goes the resolver takes care of it after that. Why it is saying the INT VPN interface is beyond me unless the traffic is getting passed there first but I wouldn't think so. The only reason I was doing it that was was to add more obscurity of the traffic on the server side. Getting connections to from a 443 that doesn't match the location of the DNS requests.
