Network problems and too many dropped TCP:PA FPA RA
-
Hello, everyone! Sorry for my bad english, i try my best :)
So, there's center office with pfsense 2.3.4 on board (VM hosted on proxmox4) with 3 directly attached networks, and about 40 remote networks connected using ipsec tunnel over providers net (not internet). In one of pfsenses lan we got zyxel usg300 gateway that gives internet to hosts in remote networks (like that https://doc.pfsense.org/index.php/Routing_internet_traffic_through_a_site-to-site_IPsec_tunnel), i've add zyxel as a additional gateway to pfsense and route all internet traffic there. and it works, but i've encountered a strange problem in result.
Customers started to complain that services do not work properly. Common example, user in network 192.168.10.0/24 (it can be ipsec net or directly attached, no matter) connected to rdp server in one of directly attached to pfsense networks, e.g, 192.168.12.250/24. Almost every 10 minutes screen turns black and user need to reconnect to rdp server. So connection to pptp vpn proxy also drops like that. Almost every net suffers from this. In firewall logs i see excessive drops by "default deny rule ipv4" with protocols like TCP:PA TCP:FPA TCP:RA
I tried turn firewall optimization into conservative mode, set fw rule to sloppy state - didn't help. I've decided to rollback config, but the problem still remains. Even rollback the VM state, but result is still the same. What really helped is adding floating rule "* * * * state none"
I already read this https://doc.pfsense.org/index.php/Why_do_my_logs_show_%22blocked%22_for_traffic_from_a_legitimate_connection, but there is too many this out-of-state traffic, so users cannot work like the old days before my experiments.Please, help me to understand the problem.
To visualize all that gibberish a couple screenshots and scheme below. Thanks!
-
192.168.6 is going to be asymmetrical and yeah will causes all kinds of problems.
You need to connect your internet router to pfsense via a transit network to remove your asymmetrical routing you have.
-
192.168.6 is going to be asymmetrical and yeah will causes all kinds of problems.
You need to connect your internet router to pfsense via a transit network to remove your asymmetrical routing you have.
alright
1. I thought it would be a problem only for 6.0/24 host trying to access internet.
correct me if i'm wrong, but when host, e.g from 192.168.10.0/24 try to connect to server in 192.168.12.0/24 this isn't be an asymmetrical. Even if 192.168.10.0/24 try to access internet it is not asymmetrical.
2. You mean i should add to pfsense another interface in 30 net and connect it to another interface in zyxel, so inetgw will be in another network? (i know zyxel gw is kinda fifth wheel, but i cannot get rid of it right now) -
Your 192.168.6 should only be connected to your pfsense.. You need to create a transit network to connect your zyxel to pfsense. Yes /30 is a normal choice for a transit network, sometimes /29 if you have a few routers that need to talk to each other to get to different networks.
In your setup 192.168.6 is transit - you do not put hosts on transit. if you do then you need to host route on them telling them where to go - or you end up with what your seeing, out of state traffic.
-
Your 192.168.6 should only be connected to your pfsense.. You need to create a transit network to connect your zyxel to pfsense. Yes /30 is a normal choice for a transit network, sometimes /29 if you have a few routers that need to talk to each other to get to different networks.
In your setup 192.168.6 is transit - you do not put hosts on transit. if you do then you need to host route on them telling them where to go - or you end up with what your seeing, out of state traffic.
As i said, host from 10.0/24 connected to server in directly attached 12.0/24 also got broken connections. it confuses me, problem affects hosts which i think do not interact with 6.0/24 network.
So, if i add interface and put it in other 30-net, should i add NAT outbound rule?
Something like this
-
Where is your nat happening? Why are you natting your ipsec traffic?
You should not nat to any of your other rfc1918 networks, you only need to nat where you go from rfc1918 to public - so you zyxel should be doing the nat for all your downstream networks.
As to your other reasons for out of state blocks, ie FA, SA, A, etc. etc.. The link you would have to look into why..
Looking at your drawing the asymmetrical issue JUMPS out - Borked setup for sure if any hosts sit on that network… Then firewall screen shot shows out of state logs for 192.168.6 address.. So yeah that screams asymmetrical problem... You do not list this dest 192.168.3 address on your drawing so not sure what would be going on there.
But putting hosts on what is a transit network is asking for problems with asymmetrical routing.. I would suggest you fix it!!! If your still seeing out of state blocks - then work out why.. Clients can cause it if they hold open a connection. Duplicate packets in your network can cause it.. FA for example - if pfsense sees this twice first time it will close the state, next packet say on a retrans would show a block because its out of state.
-
Where is your nat happening? Why are you natting your ipsec traffic?
You should not nat to any of your other rfc1918 networks, you only need to nat where you go from rfc1918 to public - so you zyxel should be doing the nat for all your downstream networks.
host accessing internet resources nated by zyxel, of course. On Pfsenses wan interface nat rules is mandatory, because there's some resources in providers net local hosts must access. So i think for transit network interface (that points to zyxel and internet at the end) also must be an outbound rule. About ipsec NAT i'm not sure, this configuration i got from a former engineer. Judgind by description, Isn't it an automatic rule?
-
No its not an automatic rule - clearly as you can see from your screenshot your in manual mode ;)
Keep in mind if you switch it back to automatic the outbound nats would stay that were put in manual mode. You normally want to clean those up, etc.
As to the providers network - why would that require nat? Unless into the providers network it wouldn't know how to get to your networks behind your routers connected to the 10 network.. If so then you would have to nat that traffic into providers network to look like its in a providers network.
The only time you need to nat when going from public to rfc1918, or if you have overlapping rfc1918 networks you need to be able to get to. Or the network you access would not know how to get to your source network.. Or if the devices in the dest network do not have a gateway, etc.
You really should avoid nat whenever possible.. For example if there is a overlap in rfc1918 space that you can work around with a nat - the better solution would be to change the networks so they do not overlap. The best thing about ipv6 is years from now when ipv4 is finally gone ;) nat will no longer be required…
-
Okay. I've add interface for transit network where zyxel is connected. I even detach link to 6.0/24 from zyxel usg :)
Aaaand…. no luck, still the same issue. Broken connections all the time, users complaining every minute, same tcp:fpa pa fa flood. In attachement several screenshots and scheme after changes. Is it still an asymmetrical issue here?upd: funny thing. internet for remote offices works well, but from yesterday i cannot do ping to internet hosts. icmp reply didn't come. While other traffic flows with no problem. I've noticed this concerns only windows hosts. From linux host i can ping whatever i want and got replies. I can see that packet leaves zyxel but response never came back. Traceroute shows last answer from providers switch.
But if i turn on outbound NAT rule on pfsense (on screenshot its turned off) ping become okay. What was that? :-\I'd appreciate any help. Thanx
-
Why are you NATTING your ipsec to the tunnel IP???
Which wan are you natting to and why? There is NO reason to nat your traffic that goes over your provider net.. The only place you should need to nat is to your internet connection. This would be natted to your transit interface you created.
And why do you have default gateway out your wan? The default gw should be out your transit to the internet.
You would have specific routes to get to your different destination via your ipsec setup… Please post up how you have your ipsec done.
So this provider net is not secure, which is why your encrypting your traffic with ipsec..
Why are you natting your transit network source to your wan?
Your nats are a mess... There really should be only the 1 nat in such a setup.. Either on your transit network to your zyxel, or on the zyxel itself to the internet. If you control the zyxel then do it there and pfsense would be doing no natting at all.
And you would want to make sure there is no natting being done on your other sites.
I would also prob rewrite your ! rules to be explicit.. With all the different networks you have setup, its possible you have some sort of overlap in the rules that can cause problems. While I am fan of the ! rules as more explicit allow statement. With all your different networks and large masks like /16 and /8 on your provider, etc. I would be very explicit in your rules... Also routing should be handled by your ipsec setup. Only time you should route out your transit to internet (it should be default) and last rule should allow this traffic from your different sources.
-
Why are you NATTING your ipsec to the tunnel IP???
Fixed! This rules really makes no sense.
Which wan are you natting to and why? There is NO reason to nat your traffic that goes over your provider net.. The only place you should need to nat is to your internet connection. This would be natted to your transit interface you created.
I have one WAN interface with address 10.39.13.6/24 (non-internet iface), through this interface works ipsec tunnels and local hosts accessing providers resources in 10.0.0.0/8 network. Without NAT on this iface local hosts cannot access, for example, 10.1.1.25. Providers no nothing about my local networks, which is why i need NAT.
New ZYXEL_INET interface serves to internet access, so as you recommend i didn't NAT it, since zyxel on the other end of link doing itAnd why do you have default gateway out your wan? The default gw should be out your transit to the internet.
If i do this i should remake my firewall rules, here is whitelist politics, to allow internet access for specific hosts.
You would have specific routes to get to your different destination via your ipsec setup… Please post up how you have your ipsec done.
There's much of it, about 40 connections. Too many screenshots i must take. Is there any console command to dump ipsec configuration?
Why are you natting your transit network source to your wan?
Sorry, don't understand. Where exactly it shows on screenshots i've posted?
I would also prob rewrite your ! rules to be explicit.. With all the different networks you have setup, its possible you have some sort of overlap in the rules that can cause problems. While I am fan of the ! rules as more explicit allow statement. With all your different networks and large masks like /16 and /8 on your provider, etc. I would be very explicit in your rules… Also routing should be handled by your ipsec setup. Only time you should route out your transit to internet (it should be default) and last rule should allow this traffic from your different sources.
From what i know, no overlaps happens. All networks on the remote points are unique. Maybe i can post my whole configuration, so you can see any mistakes? What is more preferred way to do this?
-
cannot access, for example, 10.1.1.25. Providers no nothing about my local networks, which is why i need NAT.
Your 10.x.x.x network becomes nothing more than a transit network. You are telling your different sites next hop to hit to get to network 192.168.x etc.. Your provider need has nothing to do with what you route over this transit.
Typo on my part about transit network, I mean to say why are you natting any of them to wan..
"Without NAT on this iface local hosts cannot access, for example, 10.1.1.25. Providers no nothing about my local networks, which is why i need NAT. "
What are you accessing in this provider network other than your other sites?When I get a chance - might be a few days I will throw up a example of how to do this correctly!!! You should not be natting anything to ipsec. Nor anything to your WAN which is your transit network to your other sites.. What resources other than your sites are you accessing on it?
-
What are you accessing in this provider network other than your other sites?
When I get a chance - might be a few days I will throw up a example of how to do this correctly!!! You should not be natting anything to ipsec. Nor anything to your WAN which is your transit network to your other sites.. What resources other than your sites are you accessing on it?
Providers mail server, voip gateway and some other resources besides the branches. And all of them in 10.0.0.0/8 providers network.
192.168.6.100 -> WAN -> 10.1.1.250 -> WAN -> 192.168.6.100
w/o NAT it just won't workAnyway, is it somehow related to my original problem?
-
Ok if you need to access some specifics then you would nat those say 10.1.2.3 port 80 as dest
As to your original problem - not exactly sure what your access and what causing your out of state.. You only post logs for your out of state from your 192.168.6 which was being used as your transit to your interent.
But from your drawing your 10/8 looks to be nothing more than transit - in such a case you would never need to nat to it.
So have your removed all your ipsec nats? I don't need to see every ipsec config just an example since every other site should be cookie cutter of the setup.
-
Ok if you need to access some specifics then you would nat those say 10.1.2.3 port 80 as dest
As to your original problem - not exactly sure what your access and what causing your out of state.. You only post logs for your out of state from your 192.168.6 which was being used as your transit to your interent.
I'll get some more logs tomorrow, at cost of some customers disturbance of course )
So have your removed all your ipsec nats? I don't need to see every ipsec config just an example since every other site should be cookie cutter of the setup.
For branches that do not need routing for internet traffic through pfsense each policy corresponds to specific local interface on pf. For branches that will access internet through pfsense - 0.0.0.0/0 in local network policy.
Some tunnel uses IKEv1, rest of config is the same
