[solved] Policy based routing TCP:SA
-
Hi all
I have a network with a few wans with one primary gateway for all and the others i want to point certain services to as needed so have set up the policy based routing on a PFSence using a firewall rule to match any traffic from clients IP and assign a default gateway of wan B. if i am on the client and perform a curl to a external IP site i can see the IP will change depending on if the rule is active.
if forward a port to the client and and run a tcp connection to it i can see the syn gets to the client but then i see the PFSence of wan A blocks the SA packet from leaving meaning the syn ack on the connection completely bypassed the rule and followed the default gateway on the PFsence that has the rule.
i have tried selecting all flags certain flags etc and different variations of setting withing the rule but nothing i can do seems to get the return connection to follow the rule. any sugestions of what might be causing this ?
any more info please let me know ( using latest PFsence)
-
"PFSence of wan A blocks the SA packet from leaving meaning "
How is that - so you setup floating rules? Pfsense out of the box does not block packets from leaving an interface, only from entering an interface.
A drawing of your network would be most helpful.
-
my guess was that part of the state tracking if it saw a SYN ACK without first seeing a SYN it would block it that was just a guess.
have attached a quick digram of the network involved.
no floating rules or anything each pfsence is statically configured with the default gateway of the next to point all traffic at the WANA pfsence box and that works fine but the client A box at in the diagrams needs to point out of WANB so I just created a rule as shown on PFWAN to change the gateway and direct it to WANB which worked perfectly with a curl from CLIENTA but just not on return traffic so im at a bit of a loss.
the network is inside a vmware esxi hypervisor incase that makes any diff
thanks
-
So you have your different wans no the same network?? Well thats freaking Borked right there! How did you set that up.. Pretty freaking sure pfsense would tell you NO when you created a second wan and gave it IP on the same network as another interface. Do you have them setup dhcp?
Unless are you just giving really bad examples of what your public IPs are??
So how do you have a client on 10.3.0.2 that is on a network of 192.168.1/?
Your drawing makes no sense.. So pfsense only has 1 wan IP.. Not 2 wan interfaces. And your trying to send it to a different gateway?? Which do you have as default? Post up your gateway section. And your route table from pfsense.
-
Hi sorry i think my diagram was just confusing PF WANA is a pfsence with 10.8.0.11 on the lan connected to one public IP address on the wan and WANB has 10.8.0.15 on the lan side of the pfsence and another public IP connected to the wan they are seperate wan connections with there own public ips configured on them the reason they connect to one bubble was just the software i used to create the diagram it isnt designed for network diagrams.
was just trying to give an idea of the network i have set up and that they both go into the mystical cloud that is the internet rather than they are on the same network as such.just to be clear there are 4 pf sence vm's in that diagram WANA, WANB PF-WAN and NETA are all seperate PF sence boxs the diagram is how they are phisicaly connected. with wan ip on the left and lan ip on the right
hope that clears it up a bit
-
"just to be clear there are 4 pf sence vm's in that diagram WANA,"
No it doesn't not from that drawing.. Draw your network with your actual pfsenses and the networks that connect them to their clients.
Why would you not just connect your multiple wans to 1 pfsense, and then have your networks behind that? Why do you have 4, and they are downstream of each other - so are they natting? If they are not natting where is your transit networks, etc. etc.
-
Ok added some bits to the diagram im realy not sure what else i can do to it as this is show how they are physically connected i have label all the pfSence box's wan port ip are on the left of the name LAN IP are on the right.
although not showing here there are actualy 5 wan IPs i have to make available to the network didn't think pfSence realy worked with lots of WANs connected like that may be something i need to resolve although looking at esxi i cant assign that many NICs to one box.
where there is NETA pfsence on the right there is also a NETB, NETC, NETD etc network next to it the various PFSences are being used as firewalls to controll what traffic flows between what networks and also as a router to connect the networks together.
I havent show these networks as these are not involved and allot of network to draw in that arent actualy doing anything at the moment but this is the reason for the multiple levels of PSsence box's in this and the other 3 wan IPs are there but not being used nothing is routing to them only the 2 pub IP's i have listed are actualy turned on and routing but the rest will need to come online later.
-
So your 2 pfsense behind your 2 wan pfsense boxes, are they natting? So you have a triple nat for client at 10.3.0.2 to get to the internet?
I don't see how they could be since your rule that says 10.3.02 use wana would never work, since that pfsense would never see that IP, it would only see the pfsense wan IP of 192.168.1.5
Why can you not just use 1 pfsense box with your 2 wan connections, and then put whatever networks you want behind that 1 pfsense box?
-
all natting is turned off except on those that i arrowed saying nating here so the packet keeps its source ip of 10.3.0.2 righ up untill it leaves my network hence the rule should match and dont forget i can run a curl to one of those 'whats my ip' sites and it shows WANA's public ip with the rule turned off and WANB's public ip with the rule turned on so this stuff works i have proven that its just the return sync ack when the connection is incoming that doesnt follow the rule.
id love to simpify it to your design but as mentioned right know i am only using 2 public IPS while i get this working but once i have finished this project there will be 5 public IPs and 4 local networks and i cant assign enough NICS to a virtual machine to accomodate this setup i belive the limit if 4 one lan connected to a switch to connect all the lan subnets only leaves 3 NICS for wan connection so i am 2 short, i could concolidate all the WANS to 2 pfsence machines but i would still have the problem of trying to pushing traffic 1 way or another depending on its origin and if syn acks are ignoring the firewall rules i would hit the same issue i guess putting me back to square one.
-
"setup i belive the limit if 4 one lan connected to"
Where did you get that idea?
There is no such limit.. In esxi 6 I believe there is a 10 vnic limitation. But you could also just run vlans on top of those vnics.. In hyper-v I do believe there a max of 12 vnics per vm, etc.
When you say you 5 public IPs - are those IPs from 5 different isps or 5 IPs from the same ISP?
What is the physical limitation of physical nics, in an esxi host I believe its 24 pci-e nics.. What is the speed of these isp connections. You could do it with 1 phsyical nic and all different isp could be just vlans on that 1 physical nic if your not going to surpass the physical speed limit of the physical nic.
Same goes for your lan side networks. You can use a combinations of vnics and vlans on top of those vnics, etc..
-
the idea came from the fact that the max you can select when you set it up but i see you can add more after creating.
ok so i get provided a single gig ethernet connection to my box so bandwidth is already limited here, its a trunk link with my 5 IPS that all sitting inside one vlan i just connect that physical port to a virtual switch and then i connected the 5 pfsence wan ports to that virtual switch and assigned the ip's statically so to replace this setup i see i can setup vlans against the wan link and assign them IP's so i create 5 vlans and assign them the ip's i have … now next issues i have the problem these ips are all within the same /24 range and pfsence complains that the networks are overlapping also the gateways will be the same and i only seem to be able to assign this to one of the interfaces. any work around for this :-\
-
So you have a 1 gbps connection from your isp, that gives you 5 ips.
Ok then you have actually 1 wan connection in pfsense. Your other IPs would be VIPs on this wan connection.
You can then forward inbound traffic to these different vips into your networks behind pfsense. You can then do outbound natting for specific source IPs to your specific vips.
-
Spot on !
have reconfigured it all, now running everything from one pfsence, all the subnets are now on a single lan on seperate vlans and all ips on vips and can controll routing exaly how i need to.
Thank you !!!
-
Well yeah that would be the normal way to do it ;) I have no idea what you were attempting to do other than create a train wreck ;)
Glad you got it working, KISS is your friend when setting up networks…