(2) Firewalls, (2) different networks, both mostly work, 1 can't get to a specific IP



  • Hi all -

    This one has both me and my ISP stumped.

    Firewall 1) Static IP that ends in .8 Works fine, never a problem. Can communicate over the Internet without an issue.
    Firewall 2) Static IP that ends in .7 Mostly works, many sites work fine. Fails to connect to other sites. Reports that some sites are not available for days, but occasionally pop up.

    Both Firewalls running pfSense 2.4.5-Release (amd64)
    Both Firewalls running on identical Intel Celeron low-power motherboard hardware
    Both Firewalls have fibre-channel connections to the internet via the same ISP (but are in separate buildings)
    From what I've checked (firewall advance settings), both appear to be configured the same

    Recently discovered a site that is reachable via Firewall 1 that is NOT reachable via Firewall 2: the FoldingAtHome work unit server at 65.254.110.245.

    Both pings and traceroutes fail on Firewall 2 to that IP. Both pings and traceroutes work fine on Firewall 1.

    Firewall 1 Traceroute:

    1  216.146.251.1  1.127 ms  0.893 ms  0.832 ms
     2  10.255.255.5  1.743 ms  1.378 ms  1.330 ms
     3  10.255.255.1  1.779 ms  1.725 ms  1.834 ms
     4  199.36.80.237  3.599 ms  3.553 ms  3.553 ms
     5  65.255.158.204  5.863 ms  6.301 ms  4.238 ms
     6  38.32.76.9  9.533 ms  8.299 ms  8.190 ms
     7  38.104.162.78  8.037 ms  8.148 ms  7.941 ms
     8  128.252.100.125  8.511 ms
        128.252.1.253  8.408 ms  8.529 ms
     9  128.252.161.21  8.751 ms  8.586 ms
        128.252.161.23  8.537 ms
    10  128.252.161.146  9.716 ms  9.064 ms  9.160 ms
    11  128.252.161.133  9.277 ms  9.229 ms  9.367 ms
    12  65.254.110.245  9.177 ms !Z  9.204 ms !Z  9.131 ms !Z
    

    A traceroute to that IP on Firewall 2 fails:

    1 traceroute: wrote 65.254.110.245 40 chars, ret=-1
     *traceroute: wrote 65.254.110.245 40 chars, ret=-1
     *traceroute: wrote 65.254.110.245 40 chars, ret=-1
     *
     2 traceroute: wrote 65.254.110.245 40 chars, ret=-1
     *traceroute: wrote 65.254.110.245 40 chars, ret=-1
     *traceroute: wrote 65.254.110.245 40 chars, ret=-1
     *
     3 traceroute: wrote 65.254.110.245 40 chars, ret=-1
     *traceroute: wrote 65.254.110.245 40 chars, ret=-1
     *traceroute: wrote 65.254.110.245 40 chars, ret=-1
     *
     4 traceroute: wrote 65.254.110.245 40 chars, ret=-1
     *traceroute: wrote 65.254.110.245 40 chars, ret=-1
     *traceroute: wrote 65.254.110.245 40 chars, ret=-1
    .
    .
    .
    

    and what is really weird: Here is a Traceroute on Firewall 2 (failing one) to the next-to-last hop:

    1  216.146.251.1  1.258 ms  1.172 ms  0.821 ms
     2  10.255.255.5  1.511 ms  1.336 ms  1.324 ms
     3  10.255.255.1  1.774 ms  1.891 ms  1.809 ms
     4  199.36.80.237  3.921 ms  4.060 ms  3.567 ms
     5  65.255.158.204  6.165 ms  5.968 ms  12.295 ms
     6  38.32.76.9  8.190 ms  9.083 ms  8.258 ms
     7  38.104.162.78  8.100 ms  7.856 ms  7.808 ms
     8  128.252.1.253  8.418 ms
        128.252.100.125  8.554 ms
        128.252.1.253  8.454 ms
     9  128.252.161.23  8.559 ms  8.297 ms
        128.252.161.21  8.635 ms
    10  128.252.161.146  9.274 ms  8.712 ms  8.954 ms
    11  128.252.161.133  9.285 ms *  9.176 ms
    

    ISP came out today and (bravely) plugged a laptop directly into the IP port of their fibre converter box (the same port my pfSense firewall usually plugs into). The windows box had no problems connecting to the site. I therefore kind of have to rule out the ISP and point at pfSense (which kills me to do).

    Any ideas? I'm stumped.



  • @ccgllc said in (2) Firewalls, (2) different networks, both mostly work, 1 can't get to a specific IP:

    65.254.110.245

    Pinging 65.254.110.245 with 32 bytes of data:
    Reply from 65.254.110.245: bytes=32 time=77ms TTL=43
    Reply from 65.254.110.245: bytes=32 time=71ms TTL=43
    Reply from 65.254.110.245: bytes=32 time=85ms TTL=43
    Reply from 65.254.110.245: bytes=32 time=77ms TTL=43

    Ping statistics for 65.254.110.245:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
    Minimum = 71ms, Maximum = 85ms, Average = 77ms

    No problem from here.. Maybe they are blocking you??



  • No... traceroute and ping work fine to that IP if I replace my pfSense firewall with (the ISP's) laptop. So they can't be blocking on IP.

    That and the site is the global FoldingAtHome Work Unit server - they would have no reason at all to block me.



  • What is the routing table on the firewall that doesn't work.



  • Is your pfsense actually getting the publicly available IP? Does it not change when you change devices? Or is it a static address?



  • Routing table: Works all the way to the last-to-next node, so don't think so - but do you have something specific I can check?

    Both of my Internet connections are static. Access to most sites is working fine. Just not this one I could easily check. Others, like the Black Desert Online download site might start, but will die with connection errors rather quickly. All of the problem sites, including this test case, work fine through Firewall 1.

    I need to understand what could cause a traceroute to make it 11 hops and then die on the 12th on Firewall 2, but work successfully on Firewall 1? Hops 8 & 9 look a little weird to me, but again, that shouldn't have anything to do with my firewall settings.



  • @ccgllc said in (2) Firewalls, (2) different networks, both mostly work, 1 can't get to a specific IP:

    Routing table: Works all the way to the last-to-next node, so don't think so - but do you have something specific I can check?

    I would compare the routing tables on the two devices, the fact that they're on the same subnet they should be pretty much the same. I'm thinking maybe there is an entry there that is confusing witch interface to use when going to those affected ip addresses.



  • @JohnKap said in (2) Firewalls, (2) different networks, both mostly work, 1 can't get to a specific IP:

    @ccgllc said in (2) Firewalls, (2) different networks, both mostly work, 1 can't get to a specific IP:

    Routing table: Works all the way to the last-to-next node, so don't think so - but do you have something specific I can check?

    I would compare the routing tables on the two devices, the fact that they're on the same subnet they should be pretty much the same. I'm thinking maybe there is an entry there that is confusing witch interface to use when going to those affected ip addresses.

    Routing tables are as expected:

    127.0.0.1
    The LAN port & network
    The WAN port & network

    No other entries.

    e.g. There are no "tables" I'm aware of that the firewall would build to direct traffic to a specific IP address that is not part of either its WAN or LAN group - all of those go out the default route on the WAN and passed to the next node to handle (in this case, my ISP).


Log in to reply