Dealing with asymmetric routes



  • Hi,
     the specific problem happend to me  ???

    Network topology:

    1. LAN subnet 192.168.0.0/24
      LAN IP 192.168.0.254 (em1)
      LAN ALias 192.169.0.254
           DMZ server 192.169.0.11 (IP Alias) em1

    Static route 192.168.0.0/16 via em1 - 192.168.0.252 (Provider VPN DXX SLA)
    Remote office subnets in 192.168.0.0/16 like 192.168.1.0/24; 192.168.20.0/24; 192.168.200.0/24

    1. Remote LAN's subnets 192.168.0.0/16 routed via provider router 192.168.0.252 / routes is wroking
    2. Rules - allow 192.168.0.0/16 to 192.169.0.11 (em1 rules) and 192.169.0.11 to 192.168.0.0/16 (em1 rules)

    Problem:

    1. I can ping and telnet services from 192.168.0/16 hosts to 192.169.0.11, but all tcp connections are hangs and get timeouts…
    2. I've got filter messages rule 1/0(match): block in on em1

    Jul 28 17:50:36 firewall pf: 192.169.0.11.10090 > 192.168.xx.101.62873: Flags [S.], cksum 0xb8b0 (correct), seq 1944344361, ack 506361591, win 64240, options [mss 1460,nop,wscale 0,nop,nop,sackOK], length 0
    Jul 28 17:50:36 firewall pf: 00:00:00.000160 rule 1/0(match): block in on em1: (tos 0x0, ttl 124, id 57033, offset 0, flags [DF], proto TCP (6), length 48)

    Reboots and sessions reset could not help, where is no any others blocking rules.
    Block private networks and Block bogon networks settings are not checked both on LAN and DMZ interfaces.
    Bypass firewall rules for traffic on the same interface - checked

    Any suggestions how can I troubleshot this?


  • LAYER 8 Global Moderator

    so your trying to run 2 different networks on your lan inteface? this 192.168 (private) and then a public IP 192.169?

    NetRange:      192.169.0.0 - 192.169.1.255
    OrgName:        RGnet, LLC
    OrgId:          RGNETI-1
    Address:        5147 Crystal Spring
    City:          Bainbridge Island
    StateProv:      WA

    And then your trying to route a network that your box is in 192.168/16 while you are on 192.168.0/24 to this IP that is on your own network 192.168.0.252..  so your routing your own network to this 252 box?

    is 192.169 just a typo?

    A drawing of what your trying to accomplish would be helpful..  But you don't normally route a network that includes your own network somewhere.  If you want to route 192.168/16 somewhere then you should be on a 192.168.0/24 that is inside that network.

    normally you would do something like your IP 172.16.0.1/24 to router on your network 172.16.0.2/24 for example that knows how to get to 192.168/16



  • Yes, we use 192.169. for private network from long time ago…

    CENTER with Pfsense firewall
    WAN em0 213.xxxxx
      Aliases
    LAN em1 192.168.0.254
      Alias 192.169.0.254
      Alias 10.0.0.254
      Alias more subnet
    Static routes 192.168.0.0/16 via 192.168.0.252
    Provider VPN DXX 192.168.0.252

    REMOTE offices (VPN DXX)
    Remote offices (for about 120 offices) are in provider made VPN and accessible from center by 192.168.0.0/16 subnet with follow scheme: any location has - network 192.168.xx.0/24 gw 192.168.xx.250

    Problem are above, tcp traffic cannot normally reach 192.168.0.0/24 and 192.169.0.0/24 from remote offices 192.168.xx.0/16

    So, few topics was with some similar problem with asymmetric routing and one of solution from here http://forum.pfsense.org/index.php?topic=65391.0 was to use Floating rule:

    Interface of 192.168.0.0/24 and 192.169.0.0/24 (aka em1)
    Direction any
    Protocol TCP
    Source and Destination is the created alias (alias as I see must be 192.168.0.0/16 and 192.169.0.0/24 subnets)
    TCP Flags all
    State type none

    Is it correct solution?


  • LAYER 8 Global Moderator

    "Yes, we use 192.169. for private network from long time ago… "

    So you have a jacked up setup for a long time then..  You don't just grab public space an use it internal as "private"  Are you RGnet, LLC ??

    Static routes 192.168.0.0/16 via 192.168.0.252

    So you have a locally connected network via 192.168.0.0/24 -- yet your trying to route all 192.168.0.0/16 to 192.168.0.252 -- you don't see a problem with that??



  • Ping from 192.168.0.0/24 to 192.168.xx.0/16 is OK
    But VNC from 192.168.0.0/24 to 192.168.xx.0/16 drops connection after some time…10-20 sec.
    Connection from  192.168.xx.0/16 to 192.169.0.0/24 servers drops connection after some time also...

    P.S. Everything works fine with Linux Debian firewall

    @johnpoz:

    So you have a locally connected network via 192.168.0.0/24 – yet your trying to route all 192.168.0.0/16 to 192.168.0.252 -- you don't see a problem with that??


  • Banned

    This thread is totally pointless until you've fixed the 192.169. SNAFU.  ::)



  • Why this subnet 192.169 works without probs in Linux, is this hardcode restriction?
    This subnet using just for LAN and never goes outside…

    The another problem is with normal 192.168 subnet, VNC connection drops :(

    @doktornotor:

    This thread is totally pointless until you've fixed the 192.169. SNAFU.  ::)


  • Banned

    @labasus:

    Why this subnet 192.169 works without probs in Linux, is this hardcode restriction?

    Dude. Your configuration is utterly broken:

    • you have hijacked public IP range for your LAN
    • your local and remote LANs overlap (192.168.0.0/16 includes 192.168.0.0/24)

    Go redo the network from scratch, this will never work properly.



  • I can't agree this these:

    1. public IP range in my LAN, for private use… is it problem to make it worki in Pfsense, why it works without probs in Linux... I know it is wrong, but too difficult to change on working system 24/7
    2. I have no overlaps, maybe I've skipped smth. from explanation:
      CENTER subnet 192.168.0.0/24
      REMOTE offices 192.168.xx.0/24 (sample 192.168.16.0/24, 192.168.24.0/24 and etc.)

    I just want to migrate from existing Linux firewall to PFsense, but problem with asymmetric routes make things difficult...

    How to troubleshot this, tcpdump logs, firewall logs...what else?

    @doktornotor:

    @labasus:

    Why this subnet 192.169 works without probs in Linux, is this hardcode restriction?

    Dude. Your configuration is utterly broken:

    • you have hijacked public IP range for your LAN
    • your local and remote LANs overlap (192.168.0.0/16 includes 192.168.0.0/24)

    Go redo the network from scratch, this will never work properly.


  • Banned

    You won't receive any help here for hijacking public IP ranges. It's utterly broken, fix it! Period.


  • LAYER 8 Global Moderator

    What do you mean you don't have overlap?

    LAN em1 192.168.0.254
      Alias 192.169.0.254
      Alias 10.0.0.254
      Alias more subnet
    Static routes 192.168.0.0/16 via 192.168.0.252

    That sure looks overlapped to me..

    Nobody is saying 192.169 would not work, what doktornotor is saying is is hijacked and broken.. Do you own 192.169 – No I don't think you do..  Why would you set that up??  If you didn't do it, does not matter if its a lot of work.. It should be fixed, you do not grab public IPs out of your A_S and use then internally on your network.

    What if you wanted to actually go to a site on that network?



  • So LAN 192.168.0.0/24 to 192.168.0.0/24 can reach each other without gateways and static route doesn't make sense.
    Routing is working without probs, I can reach remote host in VPN, but connection drops … - this is the real problem I want to focus at.

    Overlaping is matter than IPsec VPN is setup, but all VPN addressing and routing goes through provider Cisco routers.


  • Banned

    @labasus:

    Overlaping is matter than IPsec VPN is setup, but all VPN addressing and routing goes through provider Cisco routers.

    What? Huh? Your LANs overlap with your WAN? Even better… Excellent network design. I have an ultimate suggestion: go hire network administrator. Some sane one this time. Simple choice: whoever starts to pull their hair out once you've described your network setup is the man.  ;D ;D ;D



  • Thx for offtopic advices, these "wan in lan probs" will be fixed, be the main problem is still exist…

    @doktornotor:

    @labasus:

    Overlaping is matter than IPsec VPN is setup, but all VPN addressing and routing goes through provider Cisco routers.

    What? Huh? Your LANs overlap with your WAN? Even better… Excellent network design. I have an ultimate suggestion: go hire network administrator. Some sane one this time. Simple choice: whoever starts to pull their hair out once you've described your network setup is the man.  ;D ;D ;D


  • Banned

    @labasus:

    Thx for offtopic advices, these "wan in lan probs" will be fixed, be the main problem is still exist…

    Your main problem is that the whole thing must be redone from scratch. Why are you "debugging" something that is completely broken by design and needs to go back to design board?


  • LAYER 8 Global Moderator

    Dude I have to agree your network is broken..  We haven't even touched on this

    Alias 192.169.0.254
      Alias 10.0.0.254
      Alias more subnet

    So your running multiple address space over the same physical wire?  That is just another bad decision!

    I would hate to dig deeper..  You do understand that normally a location is given an IP range, and all segments at that location fall under this range.

    How many hosts do you have in a location?  Lets take some address space you want to work with and break that up so all your locations have more than enough addresses to work with.  Including growth!

    So for example location A is just HUGE amounts of nodes – maybe this is 192.168.0/18, you break that down into smaller segments as needed at the location.  Do they really have need for some 16k some IPs?  Next biggest location is maybe 192.168.64/20 - this gives them some 4k addresses to play with, etc.

    So you break up the total /16 giving each location a portion of the subnets of /16 to work with..  If you really have too many nodes that /16 is not enough.. then maybe you use 172.16/12 if 1Mil some addresses is not enough then use 10/8 and break that up as needed.  If need be use all 3 ranges correctly broken up this gives you almost 18Million addresses to work with..  Come on -- really you have so many hosts that the private IP space is not enough -- you have to grab a public range that you do not own to work with?  That is just BROKE!  Technically it can work, but it is BAD practice to get into!

    Now with all your locations -- how many do you have?  Lets quadruple it, hey lets x10 for growth sake..  And then use that sized subnet for your central network now the vpn connections for all your locations will have an IP in this network.  So your networks that are not local will be routed out that connection.  You could run a routing protocol, or use static routes sure - depending maybe all you need is a default route?

    Then for the different segments you have in a location, we will actually break those out to different nic vs aliases.. Or use vlans - do you have manageable or smart switches that support vlans?  Why is it your running disjointed addresses as aliases on 1 nic?

    I would really suggest you take doktornotor advice and get someone to help you if you don't understand basic networking principles..  Yeah it might be a lot of work - but from what I can tell from the info you have posted, its just seems BROKE from many different directions.

    Maybe you want to get a pfsense support contract??

    You will notice one of the things offered;
    https://portal.pfsense.org/support-subscription.php

    Network design - When deploying a new network environment, it's important to start with a sound network design. We have provided assistance with network design ranging from a review of your proposed design, to completely designing the environment to your requirements and providing complete, professional network



  • Thanks johnpoz for more delicate pushing than doktornotor… really appreciated

    Do you know the term like "lazy admin" and than you have not enough time for smth:))
    Just skipping some details and bla-bla-bla, I agree that these subnet "breaks" of couse must be fixed, and this will be first thing I really want to do in nearest future, but what about VNC connection drops from 192.168.0.0/24 center and remote 192.168.1-240.xx nods.

    Network design is a good thing if you starting to plan something and can quite enough time to test and so on, but testing on production system cannot be good idea, especially than remote offices are connected to center by provider with their own routers (managed only by provider, I have not access).

    OK, about nods count:
    CENTER - 200 (vSphere - virtual servers, workstations, printers, switches and etc)
    REMOTE offices count ~ 130 * 30 nods =  3900

    I really have enough practice and experiences in networks, hardware and in Linux administration more that 10 years and 5 years in Pfsense as also, but this network design was made before my administration.... and works like a charm, but not in Pfsense. I hope to fix some stuff to make it work on Pfsense without any hacks just like it works on Linux, later I will handle with subnets breaks....

    CENTER Pfsense box (Vmware) config
      WAN 213.xx.xx.xx
      LAN1 192.168.0.254
      LAN1 Alias 192.169.0.254 (now) ---> can be changed to LAN2 10.0.100.254
      Static route 192.168.0.0/16 via 192.168.0.252

    Of cause Vmware gives a freedom with ethernet adapter that is usable limited in physical servers, so I really can use so many virtual adapters for every ALIAS I have... OK, this is clear.

    CENTER Provider box (Cisco) -> VPN DXX service made by provider
      LAN 192.168.0.252
    Note: Out provider does not support VLAN's over VPN DXX, but I can live without it...

    Center network sample for LAN
      Network 192.168.0.0/24
      Netmask 255.255.255.0
      Gateway 192.168.0.254

    Remote office network sample from 192.168.xx.0/24
        Network  192.168.20.0/24
        Netmask 255.255.255.0
        Gateway 192.168.20.250 (provider Cisco gateway)

    From all remote offices all gateways 192.168.xx.250 are routed to the central 192.168.0.254 (these is also made by provider routers).

    All remote offices can see center 192.168.0.0/24, but not other remote offices from 192.168.1-240.0/16 (this rule made by provider by access lists in their routers).

    Johnpoz: do you still think I need to get paid support in my case?

    Is somebody had or have the similar network design and had some problems with connection drops ... lets talk about real problem, not about ideal network architechture...


  • LAYER 8 Global Moderator

    Can you draw your current setup?

    I am not seeing why you have this setup.

    CENTER Provider box (Cisco) -> VPN DXX service made by provider
      LAN 192.168.0.252

    Why is this connected to your lan?  Why is your vpn connection not a wan interface?

    And I don't see why your trying to route 192.168/16 when your on a subnet of 192.168/16

    If you vpn provider gives you an IP 192.168.0.252 on this network with a gateway of 192.168.0.254 to get to other networks.

    Here is real simple drawing..

    So you have a WAN connection in the 192.168.0/24 network – all your other locations have IPs in this network as well??  Why do you not just route directly to them..  So lets say 10.0.99/24 is at site A, your route on pfsense would say if you want to get to 10.0.99/24 talk to 192.168.0.248

    Lets say site B is 10.0.98/24 -- route that says talk to 192.168.0.249 for that network..

    Your lan network would not be on the 192.168.0/24  This network is your vpn network..

    None of the other locations would have LAN networks on this 192.168.0/24 network - it is a transient network only.  Now I am assuming your other locations all get IPs on the 192.168.0/24.. ??  What IPs do your other locations have for their vpn connections?

    This makes no sense

    Remote office network sample from 192.168.xx.0/24
        Network  192.168.20.0/24
        Netmask 255.255.255.0
        Gateway 192.168.20.250 (provider Cisco gateway)

    Is this the network they use for their LAN??  Who is providing this address space for them to use.. What if you needed a /22 at the location?  Your vpn connection should be 1 address, all of your remote locations could/would be on the same segment for this transient network.

    The issue is you don't overlap networks, and you sure don't route out a network that your currently a subnet of ;)

    I am really just making assumptions here..  And I have to head out the door right now..  But yes your network seems quite borked to me..  Unless there is something being lost in discussion.

    A drawing would be very helpful in understanding your current setup, and then how it can be converted over to using pfsense..  But again you normally would route out via a WAN connection.. In pfsense, if it has a gateway on it - its normally seen as wan and not lan.  If you have a vpn connection to other networks -- your not going to want this to be your lan network as well.




  • Since you PMed me to look at this:

    It seems like there is some good advice going around here, but there may be some salient details that get lost in writing.
    I think you should really post a diagram of your network in a form like johnpoz did. (Even if you do it in mspaint :D ).

    In general:
    1. No, there is no "hardcoded" restriction against 192.169/x in PFSense.
    2. Get rid of 192.169/16 (unless of course you are indeed RGnet).
    3. Generally don't route your own network

    However, in my opinion is would be acceptable to route a supernet of your own network as shown in the attached diagram I just drew.
    (Provided that 192.168.16.2 also has a 192.168.0/24 route via 192.168.16.1).



  • LAYER 8 Global Moderator

    Typo in your drawing there namezero or missing info?

    your routing 192.168/16 via 192.168.16.2 but you show default gateways of 192.168.16.1 and 192.168.0.1 ?  That would be bad practice as well.. You have 2 default gateways.. Yes if you have a more specfic route that route should be taken.  But your metric for your lan interface (assuming that from way your drawn) is going to be much better - so why not take that route to try and get to 192.168.2.128/25 ?

    Draw your setup up please labasus then we can all work off same picture to what your doing wrong other than the stuff already pointed out ;)



  • No, I meant 192.168.16.1 is on the other interface (upper line) connecting it via another network segment to 16.2.

    visio only lets me draw one text box per object.


  • LAYER 8 Global Moderator

    Ah that makes more sense - name of router is "default gateway"

    like this

    192.168.16.1 - router - 192.168.0.1

    You can add as many text boxes you need on a drawing




  • Yes, exactly! Apologies if the diagram was duplicitous.

    16.2 will then have more specific routes for the remote nets.

    Are those text boxes linked to the object or just "dangling" nearby?

    Either way, a diagram like that by the OP would greatly facilitate things here.



  • For example, we use something like this on the small remote sites (10-20 devices)

    Do note though that especially when you bypass FW rules for traffic on the same interface, you shouldn't have multiple subnets on the same Layer 2 segment.

    Edit: Also, if you don't want anyone talking to the VPN gateways on the VPN subnet, you should block this via firewall or alternatively move the dedicated subnet past 10.0.16.x to exclude it from the /20.




  • Here it is… network topology (to see attached files - registration required)

    If you will have some questions just ask, I can update this scheme with more details, if smth will be missed.




  • From what I understand, your original issue happens between the 192.168.0.252 MPLS router and the PFSense on the VMWare when communicating from a remote net like 192.168.1.0/24 to VM Server 192.169.0.11, correct?

    Does everything work ok when communicating between, for example, 192.168.1.0/24 and 192.168.0.0/24 (Office LAN), and the problems only happen when the IP Alias on LAN is utilized?

    Normally, the "bypass fw for subnets on same interface" should take care of the asymmetric routing for the Office LAN; That is, you only have an asymmetric route if "VM PFSense" acts as the default gateway on the network, and the "Office LAN" member has no static route to 192.168.1.0/24 defined.

    That's why I'm asking whether the problem only occurs when using the IP Alias.



  • Exactly

    @namezero111111:

    From what I understand, your original issue happens between the 192.168.0.252 MPLS router and the PFSense on the VMWare when communicating from a remote net like 192.168.1.0/24 to VM Server 192.169.0.11, correct?

    Does everything work ok when communicating between, for example, 192.168.1.0/24 and 192.168.0.0/24 (Office LAN), and the problems only happen when the IP Alias on LAN is utilized?

    Normally, the "bypass fw for subnets on same interface" should take care of the asymmetric routing for the Office LAN; That is, you only have an asymmetric route if "VM PFSense" acts as the default gateway on the network, and the "Office LAN" member has no static route to 192.168.1.0/24 defined.

    That's why I'm asking whether the problem only occurs when using the IP Alias.



  • Ok, if it only happens on the IP Alias, could you please post your /tmp/rules.debug file?
    Just sanitize the pulic IPs, they don't matter here.


Log in to reply