Dealing with asymmetric routes

labasus

I can't agree this these:

public IP range in my LAN, for private use… is it problem to make it worki in Pfsense, why it works without probs in Linux... I know it is wrong, but too difficult to change on working system 24/7
I have no overlaps, maybe I've skipped smth. from explanation:
CENTER subnet 192.168.0.0/24
REMOTE offices 192.168.xx.0/24 (sample 192.168.16.0/24, 192.168.24.0/24 and etc.)

I just want to migrate from existing Linux firewall to PFsense, but problem with asymmetric routes make things difficult...

How to troubleshot this, tcpdump logs, firewall logs...what else?

@doktornotor:

@labasus:

Why this subnet 192.169 works without probs in Linux, is this hardcode restriction?

Dude. Your configuration is utterly broken:

you have hijacked public IP range for your LAN

your local and remote LANs overlap (192.168.0.0/16 includes 192.168.0.0/24)

Go redo the network from scratch, this will never work properly.

doktornotor

You won't receive any help here for hijacking public IP ranges. It's utterly broken, fix it! Period.

johnpoz

What do you mean you don't have overlap?

LAN em1 192.168.0.254
Alias 192.169.0.254
Alias 10.0.0.254
Alias more subnet
Static routes 192.168.0.0/16 via 192.168.0.252

That sure looks overlapped to me..

Nobody is saying 192.169 would not work, what doktornotor is saying is is hijacked and broken.. Do you own 192.169 – No I don't think you do.. Why would you set that up?? If you didn't do it, does not matter if its a lot of work.. It should be fixed, you do not grab public IPs out of your A_S and use then internally on your network.

What if you wanted to actually go to a site on that network?

labasus

So LAN 192.168.0.0/24 to 192.168.0.0/24 can reach each other without gateways and static route doesn't make sense.
Routing is working without probs, I can reach remote host in VPN, but connection drops … - this is the real problem I want to focus at.

Overlaping is matter than IPsec VPN is setup, but all VPN addressing and routing goes through provider Cisco routers.

doktornotor

@labasus:

Overlaping is matter than IPsec VPN is setup, but all VPN addressing and routing goes through provider Cisco routers.

What? Huh? Your LANs overlap with your WAN? Even better… Excellent network design. I have an ultimate suggestion: go hire network administrator. Some sane one this time. Simple choice: whoever starts to pull their hair out once you've described your network setup is the man. ;D ;D ;D

labasus

Thx for offtopic advices, these "wan in lan probs" will be fixed, be the main problem is still exist…

@doktornotor:

@labasus:

Overlaping is matter than IPsec VPN is setup, but all VPN addressing and routing goes through provider Cisco routers.

What? Huh? Your LANs overlap with your WAN? Even better… Excellent network design. I have an ultimate suggestion: go hire network administrator. Some sane one this time. Simple choice: whoever starts to pull their hair out once you've described your network setup is the man. ;D ;D ;D

doktornotor

@labasus:

Thx for offtopic advices, these "wan in lan probs" will be fixed, be the main problem is still exist…

Your main problem is that the whole thing must be redone from scratch. Why are you "debugging" something that is completely broken by design and needs to go back to design board?

johnpoz

Dude I have to agree your network is broken.. We haven't even touched on this

Alias 192.169.0.254
Alias 10.0.0.254
Alias more subnet

So your running multiple address space over the same physical wire? That is just another bad decision!

I would hate to dig deeper.. You do understand that normally a location is given an IP range, and all segments at that location fall under this range.

How many hosts do you have in a location? Lets take some address space you want to work with and break that up so all your locations have more than enough addresses to work with. Including growth!

So for example location A is just HUGE amounts of nodes – maybe this is 192.168.0/18, you break that down into smaller segments as needed at the location. Do they really have need for some 16k some IPs? Next biggest location is maybe 192.168.64/20 - this gives them some 4k addresses to play with, etc.

So you break up the total /16 giving each location a portion of the subnets of /16 to work with.. If you really have too many nodes that /16 is not enough.. then maybe you use 172.16/12 if 1Mil some addresses is not enough then use 10/8 and break that up as needed. If need be use all 3 ranges correctly broken up this gives you almost 18Million addresses to work with.. Come on -- really you have so many hosts that the private IP space is not enough -- you have to grab a public range that you do not own to work with? That is just BROKE! Technically it can work, but it is BAD practice to get into!

Now with all your locations -- how many do you have? Lets quadruple it, hey lets x10 for growth sake.. And then use that sized subnet for your central network now the vpn connections for all your locations will have an IP in this network. So your networks that are not local will be routed out that connection. You could run a routing protocol, or use static routes sure - depending maybe all you need is a default route?

Then for the different segments you have in a location, we will actually break those out to different nic vs aliases.. Or use vlans - do you have manageable or smart switches that support vlans? Why is it your running disjointed addresses as aliases on 1 nic?

I would really suggest you take doktornotor advice and get someone to help you if you don't understand basic networking principles.. Yeah it might be a lot of work - but from what I can tell from the info you have posted, its just seems BROKE from many different directions.

Maybe you want to get a pfsense support contract??

You will notice one of the things offered;
https://portal.pfsense.org/support-subscription.php

Network design - When deploying a new network environment, it's important to start with a sound network design. We have provided assistance with network design ranging from a review of your proposed design, to completely designing the environment to your requirements and providing complete, professional network

labasus

Thanks johnpoz for more delicate pushing than doktornotor… really appreciated

Do you know the term like "lazy admin" and than you have not enough time for smth:))
Just skipping some details and bla-bla-bla, I agree that these subnet "breaks" of couse must be fixed, and this will be first thing I really want to do in nearest future, but what about VNC connection drops from 192.168.0.0/24 center and remote 192.168.1-240.xx nods.

Network design is a good thing if you starting to plan something and can quite enough time to test and so on, but testing on production system cannot be good idea, especially than remote offices are connected to center by provider with their own routers (managed only by provider, I have not access).

OK, about nods count:
CENTER - 200 (vSphere - virtual servers, workstations, printers, switches and etc)
REMOTE offices count ~ 130 * 30 nods = 3900

I really have enough practice and experiences in networks, hardware and in Linux administration more that 10 years and 5 years in Pfsense as also, but this network design was made before my administration.... and works like a charm, but not in Pfsense. I hope to fix some stuff to make it work on Pfsense without any hacks just like it works on Linux, later I will handle with subnets breaks....

CENTER Pfsense box (Vmware) config
WAN 213.xx.xx.xx
LAN1 192.168.0.254
LAN1 Alias 192.169.0.254 (now) ---> can be changed to LAN2 10.0.100.254
Static route 192.168.0.0/16 via 192.168.0.252

Of cause Vmware gives a freedom with ethernet adapter that is usable limited in physical servers, so I really can use so many virtual adapters for every ALIAS I have... OK, this is clear.

CENTER Provider box (Cisco) -> VPN DXX service made by provider
LAN 192.168.0.252
Note: Out provider does not support VLAN's over VPN DXX, but I can live without it...

Center network sample for LAN
Network 192.168.0.0/24
Netmask 255.255.255.0
Gateway 192.168.0.254

Remote office network sample from 192.168.xx.0/24
Network 192.168.20.0/24
Netmask 255.255.255.0
Gateway 192.168.20.250 (provider Cisco gateway)

From all remote offices all gateways 192.168.xx.250 are routed to the central 192.168.0.254 (these is also made by provider routers).

All remote offices can see center 192.168.0.0/24, but not other remote offices from 192.168.1-240.0/16 (this rule made by provider by access lists in their routers).

Johnpoz: do you still think I need to get paid support in my case?

Is somebody had or have the similar network design and had some problems with connection drops ... lets talk about real problem, not about ideal network architechture...

johnpoz

Can you draw your current setup?

I am not seeing why you have this setup.

CENTER Provider box (Cisco) -> VPN DXX service made by provider
LAN 192.168.0.252

Why is this connected to your lan? Why is your vpn connection not a wan interface?

And I don't see why your trying to route 192.168/16 when your on a subnet of 192.168/16

If you vpn provider gives you an IP 192.168.0.252 on this network with a gateway of 192.168.0.254 to get to other networks.

Here is real simple drawing..

So you have a WAN connection in the 192.168.0/24 network – all your other locations have IPs in this network as well?? Why do you not just route directly to them.. So lets say 10.0.99/24 is at site A, your route on pfsense would say if you want to get to 10.0.99/24 talk to 192.168.0.248

Lets say site B is 10.0.98/24 -- route that says talk to 192.168.0.249 for that network..

Your lan network would not be on the 192.168.0/24 This network is your vpn network..

None of the other locations would have LAN networks on this 192.168.0/24 network - it is a transient network only. Now I am assuming your other locations all get IPs on the 192.168.0/24.. ?? What IPs do your other locations have for their vpn connections?

This makes no sense

Remote office network sample from 192.168.xx.0/24
Network 192.168.20.0/24
Netmask 255.255.255.0
Gateway 192.168.20.250 (provider Cisco gateway)

Is this the network they use for their LAN?? Who is providing this address space for them to use.. What if you needed a /22 at the location? Your vpn connection should be 1 address, all of your remote locations could/would be on the same segment for this transient network.

The issue is you don't overlap networks, and you sure don't route out a network that your currently a subnet of ;)

I am really just making assumptions here.. And I have to head out the door right now.. But yes your network seems quite borked to me.. Unless there is something being lost in discussion.

A drawing would be very helpful in understanding your current setup, and then how it can be converted over to using pfsense.. But again you normally would route out via a WAN connection.. In pfsense, if it has a gateway on it - its normally seen as wan and not lan. If you have a vpn connection to other networks -- your not going to want this to be your lan network as well.

example.jpg_thumb

namezero111111

Since you PMed me to look at this:

It seems like there is some good advice going around here, but there may be some salient details that get lost in writing.
I think you should really post a diagram of your network in a form like johnpoz did. (Even if you do it in mspaint :D ).

In general:
1. No, there is no "hardcoded" restriction against 192.169/x in PFSense.
2. Get rid of 192.169/16 (unless of course you are indeed RGnet).
3. Generally don't route your own network

However, in my opinion is would be acceptable to route a supernet of your own network as shown in the attached diagram I just drew.
(Provided that 192.168.16.2 also has a 192.168.0/24 route via 192.168.16.1).

Untitled.png_thumb

johnpoz

Typo in your drawing there namezero or missing info?

your routing 192.168/16 via 192.168.16.2 but you show default gateways of 192.168.16.1 and 192.168.0.1 ? That would be bad practice as well.. You have 2 default gateways.. Yes if you have a more specfic route that route should be taken. But your metric for your lan interface (assuming that from way your drawn) is going to be much better - so why not take that route to try and get to 192.168.2.128/25 ?

Draw your setup up please labasus then we can all work off same picture to what your doing wrong other than the stuff already pointed out ;)

namezero111111

No, I meant 192.168.16.1 is on the other interface (upper line) connecting it via another network segment to 16.2.

visio only lets me draw one text box per object.

johnpoz

Ah that makes more sense - name of router is "default gateway"

like this

192.168.16.1 - router - 192.168.0.1

You can add as many text boxes you need on a drawing

Drawing1.jpg_thumb

namezero111111

Yes, exactly! Apologies if the diagram was duplicitous.

16.2 will then have more specific routes for the remote nets.

Are those text boxes linked to the object or just "dangling" nearby?

Either way, a diagram like that by the OP would greatly facilitate things here.

namezero111111

For example, we use something like this on the small remote sites (10-20 devices)

Do note though that especially when you bypass FW rules for traffic on the same interface, you shouldn't have multiple subnets on the same Layer 2 segment.

Edit: Also, if you don't want anyone talking to the VPN gateways on the VPN subnet, you should block this via firewall or alternatively move the dedicated subnet past 10.0.16.x to exclude it from the /20.

Untitled.png_thumb

labasus

Here it is… network topology (to see attached files - registration required)

If you will have some questions just ask, I can update this scheme with more details, if smth will be missed.

InternetPfsense.png_thumb

namezero111111

From what I understand, your original issue happens between the 192.168.0.252 MPLS router and the PFSense on the VMWare when communicating from a remote net like 192.168.1.0/24 to VM Server 192.169.0.11, correct?

Does everything work ok when communicating between, for example, 192.168.1.0/24 and 192.168.0.0/24 (Office LAN), and the problems only happen when the IP Alias on LAN is utilized?

Normally, the "bypass fw for subnets on same interface" should take care of the asymmetric routing for the Office LAN; That is, you only have an asymmetric route if "VM PFSense" acts as the default gateway on the network, and the "Office LAN" member has no static route to 192.168.1.0/24 defined.

That's why I'm asking whether the problem only occurs when using the IP Alias.

labasus

Exactly

@namezero111111:

From what I understand, your original issue happens between the 192.168.0.252 MPLS router and the PFSense on the VMWare when communicating from a remote net like 192.168.1.0/24 to VM Server 192.169.0.11, correct?

Does everything work ok when communicating between, for example, 192.168.1.0/24 and 192.168.0.0/24 (Office LAN), and the problems only happen when the IP Alias on LAN is utilized?

Normally, the "bypass fw for subnets on same interface" should take care of the asymmetric routing for the Office LAN; That is, you only have an asymmetric route if "VM PFSense" acts as the default gateway on the network, and the "Office LAN" member has no static route to 192.168.1.0/24 defined.

That's why I'm asking whether the problem only occurs when using the IP Alias.

namezero111111

Ok, if it only happens on the IP Alias, could you please post your /tmp/rules.debug file?
Just sanitize the pulic IPs, they don't matter here.