Dealing with asymmetric routes

labasus

Hi,
the specific problem happend to me ???

Network topology:

LAN subnet 192.168.0.0/24
LAN IP 192.168.0.254 (em1)
LAN ALias 192.169.0.254
DMZ server 192.169.0.11 (IP Alias) em1

Static route 192.168.0.0/16 via em1 - 192.168.0.252 (Provider VPN DXX SLA)
Remote office subnets in 192.168.0.0/16 like 192.168.1.0/24; 192.168.20.0/24; 192.168.200.0/24

Remote LAN's subnets 192.168.0.0/16 routed via provider router 192.168.0.252 / routes is wroking
Rules - allow 192.168.0.0/16 to 192.169.0.11 (em1 rules) and 192.169.0.11 to 192.168.0.0/16 (em1 rules)

Problem:

I can ping and telnet services from 192.168.0/16 hosts to 192.169.0.11, but all tcp connections are hangs and get timeouts…
I've got filter messages rule 1/0(match): block in on em1

Jul 28 17:50:36 firewall pf: 192.169.0.11.10090 > 192.168.xx.101.62873: Flags [S.], cksum 0xb8b0 (correct), seq 1944344361, ack 506361591, win 64240, options [mss 1460,nop,wscale 0,nop,nop,sackOK], length 0
Jul 28 17:50:36 firewall pf: 00:00:00.000160 rule 1/0(match): block in on em1: (tos 0x0, ttl 124, id 57033, offset 0, flags [DF], proto TCP (6), length 48)

Reboots and sessions reset could not help, where is no any others blocking rules.
Block private networks and Block bogon networks settings are not checked both on LAN and DMZ interfaces.
Bypass firewall rules for traffic on the same interface - checked

Any suggestions how can I troubleshot this?

johnpoz

so your trying to run 2 different networks on your lan inteface? this 192.168 (private) and then a public IP 192.169?

NetRange: 192.169.0.0 - 192.169.1.255
OrgName: RGnet, LLC
OrgId: RGNETI-1
Address: 5147 Crystal Spring
City: Bainbridge Island
StateProv: WA

And then your trying to route a network that your box is in 192.168/16 while you are on 192.168.0/24 to this IP that is on your own network 192.168.0.252.. so your routing your own network to this 252 box?

is 192.169 just a typo?

A drawing of what your trying to accomplish would be helpful.. But you don't normally route a network that includes your own network somewhere. If you want to route 192.168/16 somewhere then you should be on a 192.168.0/24 that is inside that network.

normally you would do something like your IP 172.16.0.1/24 to router on your network 172.16.0.2/24 for example that knows how to get to 192.168/16

labasus

Yes, we use 192.169. for private network from long time ago…

CENTER with Pfsense firewall
WAN em0 213.xxxxx
Aliases
LAN em1 192.168.0.254
Alias 192.169.0.254
Alias 10.0.0.254
Alias more subnet
Static routes 192.168.0.0/16 via 192.168.0.252
Provider VPN DXX 192.168.0.252

REMOTE offices (VPN DXX)
Remote offices (for about 120 offices) are in provider made VPN and accessible from center by 192.168.0.0/16 subnet with follow scheme: any location has - network 192.168.xx.0/24 gw 192.168.xx.250

Problem are above, tcp traffic cannot normally reach 192.168.0.0/24 and 192.169.0.0/24 from remote offices 192.168.xx.0/16

So, few topics was with some similar problem with asymmetric routing and one of solution from here http://forum.pfsense.org/index.php?topic=65391.0 was to use Floating rule:

Interface of 192.168.0.0/24 and 192.169.0.0/24 (aka em1)
Direction any
Protocol TCP
Source and Destination is the created alias (alias as I see must be 192.168.0.0/16 and 192.169.0.0/24 subnets)
TCP Flags all
State type none

Is it correct solution?

johnpoz

"Yes, we use 192.169. for private network from long time ago… "

So you have a jacked up setup for a long time then.. You don't just grab public space an use it internal as "private" Are you RGnet, LLC ??

Static routes 192.168.0.0/16 via 192.168.0.252

So you have a locally connected network via 192.168.0.0/24 -- yet your trying to route all 192.168.0.0/16 to 192.168.0.252 -- you don't see a problem with that??

labasus

Ping from 192.168.0.0/24 to 192.168.xx.0/16 is OK
But VNC from 192.168.0.0/24 to 192.168.xx.0/16 drops connection after some time…10-20 sec.
Connection from 192.168.xx.0/16 to 192.169.0.0/24 servers drops connection after some time also...

P.S. Everything works fine with Linux Debian firewall

@johnpoz:

So you have a locally connected network via 192.168.0.0/24 – yet your trying to route all 192.168.0.0/16 to 192.168.0.252 -- you don't see a problem with that??

doktornotor

This thread is totally pointless until you've fixed the 192.169. SNAFU. ::)

labasus

Why this subnet 192.169 works without probs in Linux, is this hardcode restriction?
This subnet using just for LAN and never goes outside…

The another problem is with normal 192.168 subnet, VNC connection drops :(

@doktornotor:

This thread is totally pointless until you've fixed the 192.169. SNAFU. ::)

doktornotor

@labasus:

Why this subnet 192.169 works without probs in Linux, is this hardcode restriction?

Dude. Your configuration is utterly broken:

you have hijacked public IP range for your LAN
your local and remote LANs overlap (192.168.0.0/16 includes 192.168.0.0/24)

Go redo the network from scratch, this will never work properly.

labasus

I can't agree this these:

public IP range in my LAN, for private use… is it problem to make it worki in Pfsense, why it works without probs in Linux... I know it is wrong, but too difficult to change on working system 24/7
I have no overlaps, maybe I've skipped smth. from explanation:
CENTER subnet 192.168.0.0/24
REMOTE offices 192.168.xx.0/24 (sample 192.168.16.0/24, 192.168.24.0/24 and etc.)

I just want to migrate from existing Linux firewall to PFsense, but problem with asymmetric routes make things difficult...

How to troubleshot this, tcpdump logs, firewall logs...what else?

@doktornotor:

@labasus:

Why this subnet 192.169 works without probs in Linux, is this hardcode restriction?

Dude. Your configuration is utterly broken:

you have hijacked public IP range for your LAN

your local and remote LANs overlap (192.168.0.0/16 includes 192.168.0.0/24)

Go redo the network from scratch, this will never work properly.

doktornotor

You won't receive any help here for hijacking public IP ranges. It's utterly broken, fix it! Period.

johnpoz

What do you mean you don't have overlap?

LAN em1 192.168.0.254
Alias 192.169.0.254
Alias 10.0.0.254
Alias more subnet
Static routes 192.168.0.0/16 via 192.168.0.252

That sure looks overlapped to me..

Nobody is saying 192.169 would not work, what doktornotor is saying is is hijacked and broken.. Do you own 192.169 – No I don't think you do.. Why would you set that up?? If you didn't do it, does not matter if its a lot of work.. It should be fixed, you do not grab public IPs out of your A_S and use then internally on your network.

What if you wanted to actually go to a site on that network?

labasus

So LAN 192.168.0.0/24 to 192.168.0.0/24 can reach each other without gateways and static route doesn't make sense.
Routing is working without probs, I can reach remote host in VPN, but connection drops … - this is the real problem I want to focus at.

Overlaping is matter than IPsec VPN is setup, but all VPN addressing and routing goes through provider Cisco routers.

doktornotor

@labasus:

Overlaping is matter than IPsec VPN is setup, but all VPN addressing and routing goes through provider Cisco routers.

What? Huh? Your LANs overlap with your WAN? Even better… Excellent network design. I have an ultimate suggestion: go hire network administrator. Some sane one this time. Simple choice: whoever starts to pull their hair out once you've described your network setup is the man. ;D ;D ;D

labasus

Thx for offtopic advices, these "wan in lan probs" will be fixed, be the main problem is still exist…

@doktornotor:

@labasus:

Overlaping is matter than IPsec VPN is setup, but all VPN addressing and routing goes through provider Cisco routers.

What? Huh? Your LANs overlap with your WAN? Even better… Excellent network design. I have an ultimate suggestion: go hire network administrator. Some sane one this time. Simple choice: whoever starts to pull their hair out once you've described your network setup is the man. ;D ;D ;D

doktornotor

@labasus:

Thx for offtopic advices, these "wan in lan probs" will be fixed, be the main problem is still exist…

Your main problem is that the whole thing must be redone from scratch. Why are you "debugging" something that is completely broken by design and needs to go back to design board?

johnpoz

Dude I have to agree your network is broken.. We haven't even touched on this

Alias 192.169.0.254
Alias 10.0.0.254
Alias more subnet

So your running multiple address space over the same physical wire? That is just another bad decision!

I would hate to dig deeper.. You do understand that normally a location is given an IP range, and all segments at that location fall under this range.

How many hosts do you have in a location? Lets take some address space you want to work with and break that up so all your locations have more than enough addresses to work with. Including growth!

So for example location A is just HUGE amounts of nodes – maybe this is 192.168.0/18, you break that down into smaller segments as needed at the location. Do they really have need for some 16k some IPs? Next biggest location is maybe 192.168.64/20 - this gives them some 4k addresses to play with, etc.

So you break up the total /16 giving each location a portion of the subnets of /16 to work with.. If you really have too many nodes that /16 is not enough.. then maybe you use 172.16/12 if 1Mil some addresses is not enough then use 10/8 and break that up as needed. If need be use all 3 ranges correctly broken up this gives you almost 18Million addresses to work with.. Come on -- really you have so many hosts that the private IP space is not enough -- you have to grab a public range that you do not own to work with? That is just BROKE! Technically it can work, but it is BAD practice to get into!

Now with all your locations -- how many do you have? Lets quadruple it, hey lets x10 for growth sake.. And then use that sized subnet for your central network now the vpn connections for all your locations will have an IP in this network. So your networks that are not local will be routed out that connection. You could run a routing protocol, or use static routes sure - depending maybe all you need is a default route?

Then for the different segments you have in a location, we will actually break those out to different nic vs aliases.. Or use vlans - do you have manageable or smart switches that support vlans? Why is it your running disjointed addresses as aliases on 1 nic?

I would really suggest you take doktornotor advice and get someone to help you if you don't understand basic networking principles.. Yeah it might be a lot of work - but from what I can tell from the info you have posted, its just seems BROKE from many different directions.

Maybe you want to get a pfsense support contract??

You will notice one of the things offered;
https://portal.pfsense.org/support-subscription.php

Network design - When deploying a new network environment, it's important to start with a sound network design. We have provided assistance with network design ranging from a review of your proposed design, to completely designing the environment to your requirements and providing complete, professional network

labasus

Thanks johnpoz for more delicate pushing than doktornotor… really appreciated

Do you know the term like "lazy admin" and than you have not enough time for smth:))
Just skipping some details and bla-bla-bla, I agree that these subnet "breaks" of couse must be fixed, and this will be first thing I really want to do in nearest future, but what about VNC connection drops from 192.168.0.0/24 center and remote 192.168.1-240.xx nods.

Network design is a good thing if you starting to plan something and can quite enough time to test and so on, but testing on production system cannot be good idea, especially than remote offices are connected to center by provider with their own routers (managed only by provider, I have not access).

OK, about nods count:
CENTER - 200 (vSphere - virtual servers, workstations, printers, switches and etc)
REMOTE offices count ~ 130 * 30 nods = 3900

I really have enough practice and experiences in networks, hardware and in Linux administration more that 10 years and 5 years in Pfsense as also, but this network design was made before my administration.... and works like a charm, but not in Pfsense. I hope to fix some stuff to make it work on Pfsense without any hacks just like it works on Linux, later I will handle with subnets breaks....

CENTER Pfsense box (Vmware) config
WAN 213.xx.xx.xx
LAN1 192.168.0.254
LAN1 Alias 192.169.0.254 (now) ---> can be changed to LAN2 10.0.100.254
Static route 192.168.0.0/16 via 192.168.0.252

Of cause Vmware gives a freedom with ethernet adapter that is usable limited in physical servers, so I really can use so many virtual adapters for every ALIAS I have... OK, this is clear.

CENTER Provider box (Cisco) -> VPN DXX service made by provider
LAN 192.168.0.252
Note: Out provider does not support VLAN's over VPN DXX, but I can live without it...

Center network sample for LAN
Network 192.168.0.0/24
Netmask 255.255.255.0
Gateway 192.168.0.254

Remote office network sample from 192.168.xx.0/24
Network 192.168.20.0/24
Netmask 255.255.255.0
Gateway 192.168.20.250 (provider Cisco gateway)

From all remote offices all gateways 192.168.xx.250 are routed to the central 192.168.0.254 (these is also made by provider routers).

All remote offices can see center 192.168.0.0/24, but not other remote offices from 192.168.1-240.0/16 (this rule made by provider by access lists in their routers).

Johnpoz: do you still think I need to get paid support in my case?

Is somebody had or have the similar network design and had some problems with connection drops ... lets talk about real problem, not about ideal network architechture...

johnpoz

Can you draw your current setup?

I am not seeing why you have this setup.

CENTER Provider box (Cisco) -> VPN DXX service made by provider
LAN 192.168.0.252

Why is this connected to your lan? Why is your vpn connection not a wan interface?

And I don't see why your trying to route 192.168/16 when your on a subnet of 192.168/16

If you vpn provider gives you an IP 192.168.0.252 on this network with a gateway of 192.168.0.254 to get to other networks.

Here is real simple drawing..

So you have a WAN connection in the 192.168.0/24 network – all your other locations have IPs in this network as well?? Why do you not just route directly to them.. So lets say 10.0.99/24 is at site A, your route on pfsense would say if you want to get to 10.0.99/24 talk to 192.168.0.248

Lets say site B is 10.0.98/24 -- route that says talk to 192.168.0.249 for that network..

Your lan network would not be on the 192.168.0/24 This network is your vpn network..

None of the other locations would have LAN networks on this 192.168.0/24 network - it is a transient network only. Now I am assuming your other locations all get IPs on the 192.168.0/24.. ?? What IPs do your other locations have for their vpn connections?

This makes no sense

Remote office network sample from 192.168.xx.0/24
Network 192.168.20.0/24
Netmask 255.255.255.0
Gateway 192.168.20.250 (provider Cisco gateway)

Is this the network they use for their LAN?? Who is providing this address space for them to use.. What if you needed a /22 at the location? Your vpn connection should be 1 address, all of your remote locations could/would be on the same segment for this transient network.

The issue is you don't overlap networks, and you sure don't route out a network that your currently a subnet of ;)

I am really just making assumptions here.. And I have to head out the door right now.. But yes your network seems quite borked to me.. Unless there is something being lost in discussion.

A drawing would be very helpful in understanding your current setup, and then how it can be converted over to using pfsense.. But again you normally would route out via a WAN connection.. In pfsense, if it has a gateway on it - its normally seen as wan and not lan. If you have a vpn connection to other networks -- your not going to want this to be your lan network as well.

example.jpg_thumb

namezero111111

Since you PMed me to look at this:

It seems like there is some good advice going around here, but there may be some salient details that get lost in writing.
I think you should really post a diagram of your network in a form like johnpoz did. (Even if you do it in mspaint :D ).

In general:
1. No, there is no "hardcoded" restriction against 192.169/x in PFSense.
2. Get rid of 192.169/16 (unless of course you are indeed RGnet).
3. Generally don't route your own network

However, in my opinion is would be acceptable to route a supernet of your own network as shown in the attached diagram I just drew.
(Provided that 192.168.16.2 also has a 192.168.0/24 route via 192.168.16.1).

Untitled.png_thumb

johnpoz

Typo in your drawing there namezero or missing info?

your routing 192.168/16 via 192.168.16.2 but you show default gateways of 192.168.16.1 and 192.168.0.1 ? That would be bad practice as well.. You have 2 default gateways.. Yes if you have a more specfic route that route should be taken. But your metric for your lan interface (assuming that from way your drawn) is going to be much better - so why not take that route to try and get to 192.168.2.128/25 ?

Draw your setup up please labasus then we can all work off same picture to what your doing wrong other than the stuff already pointed out ;)