PfSense behind another router, partial connectivity



  • I have internet connectivity via an LTE Modem+router (ZTE MF28B). I have it sitting on 192.168.0.1.
    pfSense its behind it with a WAN ip of 192.168.0.150 which is in the DMZ. Then I have LAN on 192.168.1.xxx

    I have DHCP disabled on the LTE gateway and pfSense is doing it, however Im unable to disable routing functions on the modem, as the "bridged mode" doesn't work on it (the modem is a known POS).

    The behavior Im seeing is quite strange. Windows machine on the LAN shows network connectivity, Im able to ping external ips, Skype logs in, and Im able to make calls, however Im not able to load any web pages. Additionally, when I just connect to the network, within a few seconds I can get it to load a page or 2, after which point it starts stalling again. Sites that I initially loaded would continue to function (sometimes).

    I have nothing on the network.

    What could be causing this?


  • Netgate Administrator

    Did you uncheck 'block private networks' in the WAN setup?

    Possibly a dns issue. What DNS settings are you using?

    Anything in the logs to indicate a problem?

    Mobile networks often use private subnets and large scale NAT. Perhaps you have a subnet conflict with something upstream. If you traceroute to somewhere does it show that?

    Steve



  • No I believe I didn't, I'll check that out when I have access tonight.

    DNS is set to 8.8.8.8 in pfsense, as well as on my windows machine. Im able to resolve domains just fine. Logs show nothing out of the ordinary.

    I'll try tracerouting when I get home tonight, but I doubt it has much to do with it. I tried adjusting the MTU, which seemed to have improved my ability to access websites, as in they keep working after I make the initial few requests, but any requests to NEW domains that I havent accessed before just hang there.



  • Okay, I take it back about the traceroute, this is the oddest thing I've ever seen.

    With pfSense:

    Tracing route to yahoo.com [98.138.253.109]
    over a maximum of 30 hops:
    
      1     4 ms     1 ms     1 ms  192.168.1.1
      2     3 ms     3 ms     1 ms  MF28D [192.168.0.1]
      3     *        *        *     Request timed out.
      4   114 ms   208 ms   201 ms  192.168.102.2
      5    44 ms    29 ms    36 ms  10.128.87.1
      6    51 ms    24 ms    33 ms  192.168.3.75
      7    68 ms    30 ms    34 ms  192.168.3.98
      8    49 ms    28 ms    51 ms  10.118.23.37
      9    42 ms    42 ms    27 ms  10.118.20.129
     10    40 ms    32 ms    40 ms  10.118.20.2
     11    30 ms    27 ms    32 ms  24.156.157.137
     12    23 ms    30 ms    54 ms  24.156.146.46
     13    24 ms    54 ms    51 ms  24.156.157.113
     14    40 ms    39 ms    35 ms  69.63.248.233
     15     *       99 ms   140 ms  24.156.144.178
     16     *        *        *     Request timed out.
     17    48 ms    61 ms    56 ms  ae-7.pat2.nez.yahoo.com [216.115.104.126]
     18    66 ms    60 ms    60 ms  xe-7-0-0.msr1.ne1.yahoo.com [216.115.100.5]
     19   122 ms    67 ms   105 ms  xe-5-0-0.clr2-a-gdc.ne1.yahoo.com [98.138.0.19]
    
     20    56 ms    62 ms    58 ms  et-18-25.fab8-1-gdc.ne1.yahoo.com [98.138.93.15]
    
     21    63 ms    62 ms    66 ms  po-15.bas1-7-prd.ne1.yahoo.com [98.138.240.16]
     22    60 ms    66 ms    54 ms  ir1.fp.vip.ne1.yahoo.com [98.138.253.109]
    
    Trace complete.
    
    
    
    Tracing route to forum.pfsense.org [66.219.34.171]
    over a maximum of 30 hops:
    
      1     1 ms    <1 ms     1 ms  192.168.1.1
      2     2 ms     1 ms     1 ms  MF28D [192.168.0.1]
      3     *        *        *     Request timed out.
      4   154 ms   192 ms   208 ms  192.168.102.2
      5    38 ms    37 ms    31 ms  10.128.87.9
      6    38 ms    33 ms    33 ms  192.168.3.75
      7    54 ms    46 ms    23 ms  192.168.3.98
      8    39 ms    46 ms    34 ms  10.118.23.37
      9    24 ms    40 ms    33 ms  10.118.20.129
     10    55 ms    30 ms    39 ms  10.118.20.2
     11    45 ms    55 ms    44 ms  24.156.157.137
     12    51 ms    26 ms    25 ms  24.156.146.46
     13    47 ms    42 ms    52 ms  24.156.157.113
     14    37 ms    38 ms    31 ms  69.63.248.233
     15     *       38 ms    40 ms  24.156.144.178
     16    59 ms    38 ms    49 ms  0.xe-5-2-1.pr1.chi10.tbone.rr.com [66.109.9.85]
    
     17    72 ms    74 ms    67 ms  66.109.1.67
     18   105 ms   203 ms    75 ms  ae-0-0.cr0.chi30.tbone.rr.com [66.109.6.21]
     19    61 ms    82 ms   119 ms  ae-2-0.cr0.dfw10.tbone.rr.com [66.109.6.22]
     20    70 ms    68 ms   142 ms  agg3.dllatxl301r.texas.rr.com [107.14.17.137]
     21    93 ms   172 ms    89 ms  agg1.ausutxla01r.texas.rr.com [24.175.41.47]
     22    89 ms   124 ms    86 ms  tge9-5.rdrktxaz01h.texas.rr.com [66.68.0.11]
     23   129 ms   118 ms    87 ms  xe-0-2-0-0.RDRKTXAZ1CW.tx.twcbiz.com [97.77.0.53
    ]
     24    80 ms    67 ms    87 ms  xe-1-3-0.ausxtxti1zw.tx.twcbiz.com [24.73.240.16
    3]
     25   153 ms    96 ms    83 ms  rrcs-24-227-210-130.sw.biz.rr.com [24.227.210.13
    0]
     26   101 ms    90 ms    83 ms  66.219.34.171
     27    94 ms    97 ms    98 ms  66.219.34.171
    
    Trace complete.
    
    

    No pfsense (switch plugged directly into LTE modem)

    
    Tracing route to yahoo.com [206.190.36.45]
    over a maximum of 30 hops:
    
      1     2 ms     1 ms     1 ms  MF28D [192.168.0.1]
      2     *        *        *     Request timed out.
      3    61 ms   207 ms   236 ms  192.168.102.2
      4    32 ms    25 ms    29 ms  10.128.87.5
      5    43 ms    26 ms    31 ms  192.168.3.75
      6    45 ms    43 ms    41 ms  192.168.3.98
      7    32 ms    33 ms    91 ms  10.118.23.37
      8    22 ms    36 ms    47 ms  10.118.20.129
      9    40 ms    36 ms    39 ms  10.118.20.2
     10    32 ms    39 ms    36 ms  24.156.157.137
     11    30 ms    37 ms    40 ms  24.156.146.46
     12    38 ms    42 ms    39 ms  24.156.157.113
     13    40 ms    39 ms    24 ms  69.63.248.233
     14     *        *        *     Request timed out.
     15     *        *        *     Request timed out.
     16    97 ms    76 ms    74 ms  ae-5.pat1.dnx.yahoo.com [216.115.96.34]
     17   189 ms    95 ms   145 ms  ae-6.pat1.gqb.yahoo.com [216.115.101.195]
     18   114 ms    98 ms    98 ms  ae-1.msr1.gq1.yahoo.com [66.196.67.5]
     19   102 ms    92 ms    95 ms  xe-5-0-0.clr1-a-gdc.gq1.yahoo.com [67.195.0.21]
    
     20    93 ms    95 ms   104 ms  et-17-1.fab1-1-gdc.gq1.yahoo.com [98.137.31.164]
    
     21    96 ms    99 ms    98 ms  po-16.bas1-7-prd.gq1.yahoo.com [206.190.32.27]
     22    84 ms   108 ms    97 ms  ir1.fp.vip.gq1.yahoo.com [206.190.36.45]
    
    Trace complete.
    
    
    
    Tracing route to forum.pfsense.org [66.219.34.171]
    over a maximum of 30 hops:
    
      1     1 ms     1 ms     1 ms  MF28D [192.168.0.1]
      2     *        *        *     Request timed out.
      3   226 ms   201 ms   236 ms  192.168.102.2
      4    31 ms    24 ms    37 ms  10.128.87.9
      5    22 ms    23 ms    24 ms  192.168.3.75
      6    47 ms    38 ms    39 ms  192.168.3.98
      7    28 ms    37 ms    39 ms  10.118.23.37
      8    41 ms    34 ms    38 ms  10.118.20.129
      9    32 ms    24 ms    45 ms  10.118.20.2
     10    37 ms    33 ms    37 ms  24.156.157.137
     11    41 ms    29 ms    37 ms  24.156.146.46
     12    47 ms    39 ms    38 ms  24.156.157.113
     13    43 ms    34 ms    35 ms  69.63.248.233
     14    34 ms    50 ms    44 ms  24.156.144.178
     15    43 ms    48 ms    52 ms  0.xe-5-2-1.pr1.chi10.tbone.rr.com [66.109.9.85]
    
     16    80 ms    58 ms    59 ms  66.109.1.67
     17    75 ms    59 ms    75 ms  ae-0-0.cr0.chi30.tbone.rr.com [66.109.6.21]
     18    71 ms    78 ms    60 ms  ae-2-0.cr0.dfw10.tbone.rr.com [66.109.6.22]
     19    79 ms    67 ms    75 ms  agg3.dllatxl301r.texas.rr.com [107.14.17.137]
     20    82 ms    71 ms    86 ms  agg1.ausutxla01r.texas.rr.com [24.175.41.47]
     21    91 ms    77 ms    64 ms  tge9-5.rdrktxaz01h.texas.rr.com [66.68.0.11]
     22    73 ms    76 ms    78 ms  xe-0-2-0-0.RDRKTXAZ1CW.tx.twcbiz.com [97.77.0.53
    ]
     23    84 ms   101 ms    78 ms  xe-1-3-0.ausxtxti1zw.tx.twcbiz.com [24.73.240.16
    3]
     24    93 ms    98 ms    84 ms  rrcs-24-227-210-130.sw.biz.rr.com [24.227.210.13
    0]
     25   101 ms    98 ms    89 ms  66.219.34.171
     26   100 ms    90 ms    88 ms  66.219.34.171
    
    Trace complete.
    
    

    Whats interesting, the 2nd tracert to yahoo.com shows more timeouts, yet the site loads perfectly. In the first one it doesn't load at all.



  • Im currently on this network, and forum.pfsense.org was the first domain I loaded after connection. I am now unable to load ANY site except forum.pfsense.org, which works perfectly fine. If I reconnect my connection, and load a different domain, it will be the only one I will be able to access for that session. What is going on here?


  • Netgate Administrator

    Hmm, weird indeed.
    MTU seems like a likely suspect, that can result in some websites being inaccessible, though it doesn't explain why you can get through once and then to nothing else.
    I would definitely try changing your LAN subnet to something definitely not in use somewhere in your ISPs network. Perhaps: 172.16.1.0/24
    When you are stuck accessing only one website can your still ping other sites? You seem to be able to traceroute to other places.  :-\

    Steve



  • Can you try pinging with packet size 1500 or above something in Intenret?



  • @stephenw10:

    Hmm, weird indeed.
    MTU seems like a likely suspect, that can result in some websites being inaccessible, though it doesn't explain why you can get through once and then to nothing else.
    I would definitely try changing your LAN subnet to something definitely not in use somewhere in your ISPs network. Perhaps: 172.16.1.0/24
    When you are stuck accessing only one website can your still ping other sites? You seem to be able to traceroute to other places.  :-\

    Steve

    I will try that, however  192.168.1.xx is the default LAN subnet that comes the modem. I just moved it over to pfSense. I tried using 192.168.0.xx which yielded no results.

    Yes, Im still able to ping and resolve all other sites, just not load them. I tried doing packet capture, and then accessing a site that doesn't work. I do get a response packet from the remote server as I see the response headers in the log, however it only sends one packet. Not sure where the rest of them are.

    @nothing:

    Can you try pinging with packet size 1500 or above something in Intenret?

    Says packet needs to be fragmented, thats about it.



  • I take it back about what works and doesn't. I reset MTU settings back to defaults, and I started getting more predictive behavior. Some sites just work, while others do not. Here are some that do:
    forum.pfsense.com, doc.pfsense.oeg (but not www.pfsense.org)
    google.com
    di.fm
    rona.ca
    highscalability.com
    arstecica.com (but not their static content server at cdn.arstechnica.net)

    Sites that don't work:
    arduino.cc
    stackoverflow.com
    en.wikipedia.org
    serverfault.com
    bbc.co.uk
    facebook.com

    When I traceroute them, the trace looks the same, which seems like it doesn't reach my LTE gateway at all, and gets "trapped" in pfsense.

    
    Pinging facebook.com [173.252.110.27] with 32 bytes of data:
    Request timed out.
    Request timed out.
    Reply from 192.168.99.1: Destination host unreachable.
    Reply from 192.168.99.1: Destination host unreachable.
    
    Ping statistics for 173.252.110.27:
        Packets: Sent = 4, Received = 2, Lost = 2 (50% loss),
    
    C:\Users\Gerty>tracert facebook.com
    
    Tracing route to facebook.com [173.252.110.27]
    over a maximum of 30 hops:
    
      1     1 ms     1 ms     1 ms  192.168.99.1
      2  192.168.99.1  reports: Destination host unreachable.
    
    Trace complete.
    
    Tracing route to bbc.co.uk [212.58.253.67]
    over a maximum of 30 hops:
    
      1     1 ms     1 ms     1 ms  192.168.99.1
      2     *        *        *     Request timed out.
      3     *        *     192.168.99.1  reports: Destination host unreachable.
    
    Trace complete.
    
    Tracing route to arduino.cc [174.129.243.245]
    over a maximum of 30 hops:
    
      1     1 ms     1 ms     1 ms  192.168.99.1
      2     *        *        *     Request timed out.
      3     *        *     192.168.99.1  reports: Destination host unreachable.
    
    Trace complete.
    
    Tracing route to stackoverflow.com [198.252.206.16]
    over a maximum of 30 hops:
    
      1     1 ms     2 ms     1 ms  192.168.99.1
      2     *        *        *     Request timed out.
      3     *        *     192.168.99.1  reports: Destination host unreachable.
    
    Trace complete.
    
    

    I moved my LAN to 192.168.99.0/24 subnet.



  • Ok, found the issue. I had the WAN interface set to 192.168.0.150/1 instead of /24. As soon as I changed that, everything started working.


Log in to reply